Sensing assembly for autonomous driving

ABSTRACT

An autonomous driving assembly for a vehicle includes a plurality of lidar groups supported by a vehicle body of the vehicle and collectively configured to detect a periphery region in proximity to the vehicle body. Different ones of the plurality of lidar groups are supported at different areas of the vehicle body and have different group fields of view. At least two of the different group fields of view overlap with each other. Each of the plurality lidar groups includes a plurality of lidar units fixed at a same location. Different ones of the plurality of lidar units have different unit fields of view. At least two of the different unit fields of view overlap with each other.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/664,204, filed on Oct. 25, 2019, which is a continuation ofInternational Application No. PCT/CN2017/082606, filed Apr. 28, 2017,the entire contents of both of which are incorporated herein byreference.

BACKGROUND

Autonomous vehicles are designed to sense and navigate through anenvironment without guidance from a human controller. Examples ofautonomous vehicles include “self-driving” automobiles that are capableof driving without (or with minimal) human intervention. Automaticallydriving systems can detect an environment of a vehicle and allowautonomous navigation of the vehicle using multiple sensors. Varioustypes of sensors can be used to generate positional and motioninformation enabling control and navigation of an autonomous vehicle.For example, some autonomous driving systems utilize a variety ofsensors such as the Velodyne 64-channel lidar to achieve environmentaldetection.

However, current sensing systems for autonomous piloting of vehicles canbe less than ideal. Current sensing systems such as the Velodyne64-channel lidar are costly and may not have redundancies built-in. Forexample, such systems may be not be capable of determining which sensorsto utilize, particularly when one or more sensors are malfunctioning orgenerating inaccurate data under different environmental conditions. Thelimited abilities of current sensing systems may reduce their usefulnessand potentially comprise the safety of autonomous vehicles when thevehicles are self-piloting through certain types of environments.

SUMMARY

A need exists for improved autonomous driving systems. In someembodiments, the autonomous driving systems can (1) enable seamlessenvironmental sensing in 360 degrees, (2) provide redundant, reliableand stable environment sensing data, and/or (3) effect an easy and quicksensor calibration and a real-time calibration verification. In somecases, the autonomous driving systems can be configured to select whichsensors and/or data to utilize under various driving and/orenvironmental conditions.

The present disclosure addresses this need and provides relatedadvantages as well. For example, the embodiments described herein canenhance flexibility and accuracy autonomous driving systems for vehiclessuch as automobiles. In particular, the disclosed methods and systemscan improve the operational safety of vehicles, and enable thesevehicles to be self-piloted in a safe manner.

An aspect of the disclosure is directed to an autonomous drivingassembly for a vehicle, said assembly comprising: a plurality of lidarunits configured to be supported by a vehicle body, said lidar unitscollectively configured to detect a periphery region in proximity to thevehicle body to aid in autonomous driving upon coupling said drivingassembly to the vehicle body, wherein each of the plurality of lidarunits has a field of view of less than about 180 degrees.

In some embodiments, the plurality of lidar units comprise a firstsubset of lidar units comprising at least two lidar units having a fixeddisposition relative to one another, and a second subset of lidar unitscomprising at least two lidar units having a fixed disposition relativeto one another. In some embodiments, the first subset of lidar units andthe second subset of lidar units are supported on different areas of thevehicle body, and configured to work in concert to detect said region ora portion thereof. In some embodiments, the region detectable by theplurality of lidar units extends around the vehicle body in 360 degrees.In some embodiments, the first subset of lidar units is supported by afirst support structure, and the second subset of lidar units issupported by a second support structure that is separate from the firstsupport structure. In some embodiments, each of the plurality of lidarunits is a single channel lidar unit. In some embodiments, at least oneof the plurality of lidar units is a multi-channel lidar unit. In someembodiments, the plurality of lidar units are not part of amulti-channel monolithic lidar unit.

In some embodiments, the first subset of lidar units comprises a firstlidar unit optically aligned in a first direction and a second lidaroptically aligned in a second direction, wherein an angle between thefirst direction and the second direction is about 50 degrees or less. Insome embodiments, each of the plurality of lidar units has a field ofview less than about 90 degrees. In some embodiments, each of theplurality of lidar units has a field of view of about 60 degrees. Insome embodiments, the plurality of lidar units comprise less than orequal to about 12 lidar units.

In some embodiments, the first subset of lidar units is located at afirst corner of the vehicle and the second subset of lidar units islocated at a second corner of the vehicle. In some embodiments, thefirst and second corners are located on a same side of the vehicle. Insome embodiments, the first and second corners are located on differentsides of the vehicle. In some embodiments, the first and second cornersare located on opposite sides of the vehicle. In some embodiments, thefirst and second corners are located on adjacent sides of the vehicle.In some embodiments, the plurality of lidar units further comprise: athird subset of lidar units comprising at least two lidar unitssupported by a third support structure and a fourth subset of lidarunits comprising at least two lidar units supported by a fourth supportstructure. In some embodiments, the third subset of lidar units islocated at a third corner of the vehicle and the fourth subset of lidarunits is located at a fourth corner of the vehicle. In some embodiments,the third and fourth corners of the vehicle are located opposite to thefirst and second corners of the vehicle. In some embodiments, the firstsubset of lidar units may be primarily oriented facing outward along afirst diagonal from the first corner of the vehicle, the second subsetof lidar units may be primarily oriented facing outward along a seconddiagonal from the second corner of the vehicle, the third subset oflidar units may be primarily oriented facing outward along a thirddiagonal from a third corner of the vehicle, and the fourth subset oflidar units may be primarily oriented facing outward along a fourthdiagonal from a fourth corner of the vehicle.

In some embodiments, the first and second subsets of lidar units arelocated on a same side or different sides of the vehicle. In someembodiments, the first subset of lidar units is located on a first sideof the vehicle and the second subset of lidar units is located on asecond side of the vehicle. In some embodiments, the first and secondsides of the vehicle are adjacent to each other. In some embodiments,the first and second sides of the vehicle are opposite to each other. Insome embodiments, the plurality of lidar units further comprise: a thirdsubset of lidar units comprising at least two lidar units supported by athird support structure and a fourth subset of lidar units comprising atleast two lidar units supported by a fourth support structure. In someembodiments, the third subset of lidar units is located on a third sideof the vehicle and the fourth subset of lidar units is located on afourth side of the vehicle. In some embodiments, at least two of thefirst, second, third and fourth sides are located on opposite sides ofthe vehicle. In some embodiments, wherein at least two of the first,second, third and fourth sides are located on adjacent sides of thevehicle.

In some embodiments, the first subset of lidar units is primarilyoriented in a first direction facing away from the vehicle, the secondsubset of lidar units is primarily oriented in a second direction facingaway from the vehicle. In some embodiments, the plurality of lidar unitsfurther comprise: a third subset of lidar units comprising at least twolidar units supported by a third support structure and a fourth subsetof lidar units comprising at least two lidar units supported by a fourthsupport structure. In some embodiments, the third subset of lidar unitsis primarily oriented in a third direction facing away from the vehicle,and the fourth subset of lidar units is primarily oriented in a fourthdirection facing away from the vehicle. In some embodiments, two or moreof the first, second, third, and fourth directions are orthogonal toeach other. In some embodiments, two or more of the first, second,third, and fourth directions are parallel to each other. In someembodiments, two or more of the first, second, third, and fourthdirections are oblique to each other.

In some embodiments, the first subset of lidar units comprises at leastthree lidar units supported by the first support structure. In someembodiments, the at least three lidar units of the first subset of lidarunits are arranged in a manner to increase overlap between adjacentfields of view of said lidar units. In some embodiments, the secondsubset of lidar units comprises at least three lidar units supported bythe second support structure. In some embodiments, the at least threelidar units of the second subset of lidar units are arranged in a mannerto increase overlap between adjacent fields of view of said lidar units.In some embodiments, the first subset of lidar units is configured tomove relative to the second subset of lidar units so as to adjust anoverlap of field of view therebetween. In some embodiments, the firstsubset of lidar units and the second subset of lidar units areconfigured to move relative to each other so as to adjust an overlap offield of view therebetween. In some embodiments, an overlap of field ofview between the first subset of lidar units and the second subset oflidar units is adjustable in real-time to compensate for blind spotswhile the vehicle is in operation. In some embodiments, at least 70degrees of overlap of field of view exists between the first subset oflidar units and the second subset of lidar units. In some embodiments,the first subset of lidar units comprises a collective field of view ofat least 160 degrees, and the second subset of lidar units comprises acollective field of view of at least 160 degrees.

In some embodiments, a collective field of view of the first subset oflidar units is adjustable in real-time by a changing a position of atleast one lidar unit in the first subset of lidar units while thevehicle is in operation. In some embodiments, a collective field of viewof the second subset of lidar units is adjustable in real-time by achanging a position of at least one lidar unit in the second subset oflidar units while the vehicle is in operation. In some embodiments, acollective field of view of the first and second subsets of lidar unitsis adjustable in real-time by a changing a position of at least one ofthe first and second subsets of lidar units while the vehicle is inoperation. In some embodiments, the collective field of view of thefirst and second subsets of lidar units is adjustable by changing thepositions of the first and second subsets of lidar units relative toeach other. In some embodiments, the collective field of view of thefirst and second subsets of lidar units is adjustable by changing theposition of the first subsets of lidar units relative to the secondsubset of lidar units.

In some embodiments, the collective field of view of the first andsecond subsets of lidar units is inversely proportional to a collectivedetection range of the first and second subsets of lidar units. In someembodiments, an increase in the collective field of view of the firstand second subsets of lidar units causes the collective detection rangeof the first and second subsets of lidar units to decrease. In someembodiments, the collective field of view and the collective detectionrange of the first and second subsets of lidar units are adjustable inreal-time while the vehicle is in operation, depending on a width of thecollective field of view that is being selected, and/or depending on adistance of the collective detection range that is being selected.

In some embodiments, each lidar unit of the first subset of lidar unitsis fixedly attached to the first support structure, and each lidar unitof the second subset of lidar units is fixedly attached to the secondsupport structure. In some embodiments, the at least two lidar units ofthe first subset of lidar units are configured to not move relative toone another during operation of the vehicle, and wherein the at leasttwo lidar units of the second subset of lidar units are configured tonot move relative to one another during operation of the vehicle. Insome embodiments, each of the plurality of lidar units is configured toremain at a fixed position relative to the vehicle body during operationof the vehicle. In some embodiments, the fixed disposition between theat least two lidar units of the first subset of lidar units ismaintained with aid of a fixture device configured to rigidly affix saidlidar units.

In some embodiments, the fixed disposition between the at least twolidar units of the first subset of lidar units is maintained with aid ofa carrier. In some embodiments, the carrier is configured to permitmovement in one or more degrees of freedom so as to maintain the fixeddisposition between the at least two lidar units of the first subset oflidar units. In some embodiments, the carrier comprises a single-axisgimbal or a multi-axis gimbal. In some embodiments, the carrier isconfigured to adjust a position of at least one of the two lidar unitsof the first subset of lidar units, so as to maintain the fixeddisposition between the at least two lidar units. In some embodiments,the carrier is configured to adjust said position of the at least one ofthe two lidar units in real-time during operation of the vehicle. Insome embodiments, the fixed disposition between the at least two lidarunits of the first subset of lidar units is maintained with aid of oneor more linkages. In some embodiments, the linkages include serialand/or parallel linkages. In some embodiments, the fixed dispositionbetween the at least two lidar units of the first subset of lidar unitsis maintained with aid of a kinematic coupling. In some embodiments, thefixed disposition between the at least two lidar units of the firstsubset of lidar units is maintained by mechanically coupling said lidarunits in a rigid manner.

In some embodiments, at least one of the plurality of lidar units isconfigured to move relative to the vehicle body during operation of thevehicle. In some embodiments, the at least one of the plurality of lidarunits is configured to move relative to the vehicle body with aid of acarrier. In some embodiments, the first subset of lidar units isconfigured to move relative to the second subset of lidar units to focuson a predefined portion of said region while the vehicle is inoperation. In some embodiments, the first subset of lidar units and thesecond subset of lidar units are configured to move relative to eachother to focus on a predefined portion of said region while the vehicleis in operation. In some embodiments, the first subset of lidar unitsand the second subset of lidar units are configured to move relative toeach other with aid of one or more carriers.

In some embodiments, the predefined portion of said region has adifferent object density than rest of said region. In some embodiments,the predefined portion of said region has a higher object density thanrest of said region.

In some embodiments, the assembly further comprises: a long range lidarunit comprising a field of view that is narrower than a collective fieldof view of the plurality of lidar units. In some embodiments, the fieldof view of the long range lidar is narrower than a collective field ofview of the first subset of lidar units or the second subset of lidarunits. In some embodiments, the field of view of the long range lidar isgreater than a collective field of view of the first subset of lidarunits or the second subset of lidar units. In some embodiments, the longrange lidar unit comprises a greater distance range than the pluralityof lidar units. In some embodiments, the long range lidar unit has aprimary direction facing the front of the vehicle.

In some embodiments, the first and/or second subsets of lidar unitsundergoes an initial intrinsic calibration prior to utilization of thefirst and/or second subsets of lidar units for sensing. In someembodiments, the lidar units within the first subset do not requireonline calibration during operation of the vehicle, and the lidar unitswithin the second subset do not require online calibration duringoperation of the vehicle. In some embodiments, the first subset of lidarunits and the second subset of lidar units undergo an online calibrationrelative to each other during the operation of the vehicle.

An additional aspect of the disclosure is directed to a vehiclecomprising the autonomous driving assembly as previously described. Insome embodiments, the vehicle is a land-bound vehicle. In someembodiments, the vehicle comprises space for one or more passengers. Insome embodiments, the vehicle comprises one or more additional sensorsconfigured to collect information about an environment around thevehicle. In some embodiments, the one or more additional sensorscomprise one or more of the following: vision sensor, ultrasonic sensor,GPS, or wheel odometer. In some embodiments, the one or more additionalsensors comprises a millimeter wave radar. In some embodiments, theinformation from the one or more additional sensors is combined withinformation from the plurality of lidar units to aid in autonomousoperation of the vehicle. In some embodiments, the one or moreadditional sensors are calibrated to at least one of the first or secondsubsets of lidar units.

Further aspects of the disclosure are directed to a method of collectinginformation around a vehicle for autonomous driving, said methodcomprising: supporting, with aid of a vehicle body, a plurality of lidarunits of an autonomous driving assembly for the vehicle, said lidarunits collectively configured to detect a periphery region in proximityto the vehicle body to aid in autonomous driving upon coupling saiddriving assembly to the vehicle body, wherein each of the plurality oflidar units has a field of view of less than 180 degrees.

In some embodiments, the plurality of lidar units comprise a firstsubset of lidar units and a second subset of lidar units, the methodfurther comprising: obtaining data using the first subset of lidar unitscomprising at least two lidar units having a fixed disposition relativeto one another, and obtaining data using the second subset of lidarunits comprising at least two lidar units having a fixed dispositionrelative to one another, wherein the first subset of lidar units and thesecond subset of lidar units are supported on different areas of thevehicle body, and configured to work in concert to detect said region ora portion thereof.

In accordance with additional aspects of the disclosure, a sensingsystem to aid in autonomous operation of a vehicle is provided, saidsensing system comprising: a plurality of sensors configured to besupported by a vehicle body, wherein said plurality of sensors comprise:(1) a first set of sensors comprising two or more different types ofsensors oriented in a forward-facing direction and configured to detecttwo or more regions in front of the vehicle, and (2) a second set ofsensors comprising one or more types of sensors oriented in a pluralityof directions and configured to detect one or more regions in proximityto or surrounding the vehicle, wherein a range of each of the two ormore regions in front of the vehicle extends farther away from thevehicle compared to a range of each of the one or more regions inproximity to or surrounding the vehicle.

In some embodiments, the first set of sensors is configured to beoriented in the forward-facing direction while the vehicle is moving inthe forward direction. In some embodiments, at least one sensor from thefirst set of sensors is configured to change its orientation based onthe vehicle's motion or predicted motion path. In some embodiments, theat least one type of sensor from the first set of sensors is configuredto change its orientation in real-time to preemptively scan forobstacles prior to or as the vehicle is changing its motion or predictedmotion path. In some embodiments, the at least one type of sensor fromthe first set of sensors is configured to change its orientation byrotating a predetermined amount based on the vehicle's motion orpredicted motion path. In some embodiments, the at least one type ofsensor from the first set of sensors is configured to rotate clockwiseprior to or as the vehicle changes its direction to the right, so as todetect a region to the front-right of the vehicle. In some embodiments,the at least one type of sensor from the first set of sensors isconfigured to rotate counterclockwise prior to or as the vehicle changesits direction to the left, so as to detect a region to the front-left ofthe vehicle. In some embodiments, an angle of rotation of the at leastone type of sensor from the first set of sensors is adjusted based on aturn angle or arc length in the vehicle's motion or predicted motionpath. In some embodiments, the predicted motion path is predicted basedon a vehicle input and/or a map of an environment where the vehicle islocated. In some embodiments, said vehicle input comprises a drivingroute between a start point and a destination. In some embodiments, saidvehicle input comprises activation of a turn signal of the vehicle. Insome embodiments, said vehicle input comprises a rotation of a steeringwheel of the vehicle. In some embodiments, said vehicle input comprisesa change in direction of one or more driving wheels of the vehicle.

In some embodiments, the two or more different types of sensors in thefirst set of sensors are selected from the group consisting of amonocular camera, a long range lidar unit, and a millimeter-wavelengthradar unit. In some embodiments, the first set of sensors furthercomprises one or more types of sensors oriented in a backward-facingdirection. In some embodiments, the one or more types of sensorsoriented in the backward-facing direction are selected from the groupconsisting of a monocular camera, a long range lidar unit, and amillimeter-wavelength radar unit. In some embodiments, the first set ofsensors comprises more types of sensors that are oriented in theforward-facing direction than the backward-facing direction.

In some embodiments, the forward-facing monocular camera is configuredhaving a higher imaging resolution than the backward-facing monocularcamera. In some embodiments, the forward-facing monocular camera has a4K imaging resolution and the backward-facing monocular camera has a1080p imaging resolution.

In some embodiments, the two or more regions in front of the vehicleoverlap with one another. In some embodiments, the two or more regionsin front of the vehicle comprise (1) a first region detectable by afirst type of sensor selected from the first set of sensors and (2) asecond region detectable by a second type of sensor selected from thefirst set of sensors. In some embodiments, the first detectable regionlies completely within the second detectable region. In someembodiments, a portion of the first detectable region lies within thesecond detectable region, and another portion of the first detectableregion lies outside of the second detectable region. In someembodiments, the first detectable region and the second detectableregion have different ranges. In some embodiments, a range of the seconddetectable region is greater than a range of the first detectableregion. In some embodiments, an area or volume of the first detectableregion is determined by a scan angle of the first type of sensor, and anarea or volume of the second detectable region is determined by a scanangle of the second type of sensor. In some embodiments, the scan angleof the first type of sensor is greater than the scan angle of the secondtype of sensor. In some embodiments, a detection range of the first typeof sensor is less than a detection range of the second type of sensor.In some embodiments, the detection range of at least one of the firstand second types of sensors is at least 180 m. In some embodiments, allof the sensors from the first set of sensors are operational andactively detecting the two or more regions in front of the vehicle whilethe vehicle is moving forward.

In some embodiments, the first type of sensor is more suitable for usein a first type of environment, and the second type of sensor is moresuitable for use in a second type of environment. In some embodiments,the first type of environment comprises one or more of the followingelements: rain, snow, fog, and heavy dust. In some embodiments, thefirst and second types of environment have different lightingconditions. In some embodiments, the first and second types ofenvironment comprises different object densities, different types ofobjects, and/or different sizes of objects. In some embodiments, thefirst and second types of environment have different visibility ranges.In some embodiments, the first type of sensor is configured to activelydetect the first region and the second type of sensor is configured tobe passive or inactive while the vehicle is moving through or about tomove through the first type of environment. In some embodiments, thesecond type of sensor is configured to actively detect the second regionand the first type of sensor is configured to be passive or inactivewhile the vehicle is moving through or about to move through the secondtype of environment. In some embodiments, the first and second types ofsensors are configured to collect data as the vehicle is moving throughthe first and second types of environment. In some embodiments, the datafrom the first type of sensor is processed, and the data from the secondtype of sensor is not processed, as the vehicle is moving through thefirst type of environment. In some embodiments, the data from the secondtype of sensor is processed, and the data from the first type of sensoris not processed, as the vehicle is moving through the second type ofenvironment.

In some embodiments, the first set of sensors are rigidly coupled to thevehicle body. In some embodiments, the first set of sensors are movablycoupled to the vehicle body. In some embodiments, the first set ofsensors are movable relative to the vehicle body with aid of one or morecarriers. In some embodiments, the one or more carriers are configuredto permit movement of the first set of sensors about one or more degreesof freedom. In some embodiments, the first set of sensors comprises atleast one type of sensor rigidly coupled to the vehicle body and atleast one other type of sensor movably coupled to the vehicle body. Insome embodiments, the two or more different types of sensors from thefirst set of sensors are coupled adjacent to one another in a lateralconfiguration on the vehicle body. In some embodiments, the two or moredifferent types of sensors from the first set of sensors are coupledadjacent to one another in a vertical configuration on the vehicle body.

In some embodiments, the one or more different types of sensors in thesecond set of sensors are selected from the group consisting of stereocameras, lidar units, and ultrasonic sensors. In some embodiments, thesecond set of sensors comprise a plurality of stereo cameras and aplurality of lidar units. In some embodiments, the plurality of stereocameras are configured to capture color image and depth data. In someembodiments, data collected by the plurality of stereo cameras and datacollected by the plurality of lidar units from the second set of sensorsare fused together to generate a set of RGB-D data that isrepresentative of a 3D color map of a region in proximity to orsurrounding the vehicle. In some embodiments, the RGB-D data is usableto detect the presence and type of obstacles in a region in proximity toor surrounding the vehicle. In some embodiments, the RGB-D data is fusedwith data from other types of sensors from the first and/or second setsof sensors to extract more details about a region in proximity to orsurrounding the vehicle. In some embodiments, data collected by theplurality of stereo cameras is used for obstacle detection and forgenerating a first set of obstacle information, wherein data collectedby the plurality of lidar units is used for obstacle detection and forgenerating a second set of obstacle information, and wherein the firstand second sets of obstacle information are fused together to generatean environmental map of a region in proximity to or surrounding thevehicle. In some embodiments, different weight values are assigned tothe first and second sets of obstacle information depending on avisibility factor of a region in proximity to or surrounding thevehicle. In some embodiments, the visibility factor is determined basedon the data collected by the plurality of stereo cameras. In someembodiments, the first set of obstacle information is assigned a higherweight value than the second set of obstacle information when thevisibility factor is above a predetermined threshold. In someembodiments, the first set of obstacle information is assigned a lowerweight value than the second set of obstacle information when thevisibility factor is below the predetermined threshold.

In some embodiments, the plurality of stereo cameras comprise aplurality of vision sensors supported on multiple sides of the vehiclebody. In some embodiments, the plurality of vision sensors is configuredto collect data from four sides around the vehicle body. In someembodiments, the plurality of vision sensors are configured to becombined in different ways to form different stereo cameras. In someembodiments, the plurality of vision sensors are combinable to form amulti-ocular vehicular surround vision system. In some embodiments, theplurality of vision sensors comprise: (1) a first subset of visionsensors comprising at least two vision sensors having a first baselinethat collectively form a first stereo camera, and (2) a second subset ofvision sensors comprising at least two vision sensors having a secondbaseline shorter than the first baseline and that collectively form asecond stereo camera. In some embodiments, the first stereo camera has afarther visual detection range than the second stereo camera. In someembodiments, the first stereo camera has a wider field of view than thesecond stereo camera. In some embodiments, the second stereo camera isconfigured to detect a first region that falls outside the field of viewof the first stereo camera. In some embodiments, the first regioncorresponds to a blind spot of the first stereo camera. In someembodiments, the first region is closer to the vehicle body than asecond region that falls within the field of view of the first stereocamera. In some embodiments, the first stereo camera and the secondstereo camera utilize at least one common vision sensor. In someembodiments, each of the first stereo camera and the second stereocamera comprises a unique pair of vision sensors. In some embodiments, apair of vision sensors of the second stereo camera is positioned betweena pair of vision sensors of the first stereo camera on at least one sideof the vehicle body. In some embodiments, the second stereo cameracomprises (1) a first vision sensor positioned between a pair of visionsensors of the first stereo camera, and (2) a second vision sensor thatis not positioned between the pair of vision sensors of the first stereocamera. In some embodiments, the first subset of vision sensors and thesecond subset of vision sensors are on a first side of the vehicle, andthe sensing system further comprises a third subset of vision sensorscomprising at least two vision sensors collectively configured tocollect data from a second side of the vehicle and a fourth subset ofvision sensors comprising at least two vision sensors collectivelyconfigured to collect data from the second side of the vehicle, whereinthe third subset and the fourth subset of vision sensors utilize atleast one common vision sensor. In some embodiments, the second side ofthe vehicle is opposite the first side of the vehicle. In someembodiments, the sensing system further comprises a fifth subset ofvision sensors comprising at least two vision sensors on a third side ofthe vehicle and collectively configured to collect data from the thirdside of the vehicle. In some embodiments, the third side of the vehicleis a front side of the vehicle. In some embodiments, the fifth subset ofvision sensors does not utilize any common vision sensors with the firstsubset or second subset of vision sensors. In some embodiments, thesensing system is configured to collect data from at least three sidesaround the vehicle body with aid of 8 vision sensors or less. In someembodiments, the sensing system is configured to collect data from atleast 225 degrees around the vehicle body with aid of 8 vision sensorsor less.

In some embodiments, each type of sensor in the first and second sets ofsensors is configured to obtain and automatically transmit data directlyto a sensing module subscribed to the corresponding type of sensor. Insome embodiments, the plurality of types of sensors in the first andsecond sets of sensors are configured to obtain the data in anasynchronous manner. In some embodiments, the plurality of types ofsensors in the first and second sets of sensors are configured to obtainthe data at different frequencies. In some embodiments, the dataobtained by the plurality of types of sensors is synchronized and/orcalibrated at predetermined time intervals. In some embodiments, two ormore types of sensors in the first and second sets of sensors areconfigured to obtain and automatically transmit data to a sensor fusionmodule for fusing together said data.

In some embodiments, the second set of sensors comprises a plurality ofultrasonic sensors supported on multiple sides of the vehicle body. Insome embodiments, the plurality of ultrasonic sensors are configured todetect objects independent of visual characteristics of said objects,wherein said visual characteristics include a color, reflectivity,and/or texture of said objects. In some embodiments, the plurality ofultrasonic sensors are configured to detect objects that are not capableof being detected by cameras in the first and second sets of sensors. Insome embodiments, at least one of the plurality of ultrasonic sensors ismovable relative to the vehicle body with aid of a carrier. In someembodiments, the at least one of the plurality of ultrasonic sensors ismovable about one or more degrees of freedom with aid of the carrier. Insome embodiments, the carrier comprises a single axis gimbal or amulti-axis gimbal. In some embodiments, the at least one of theplurality of ultrasonic sensors is configured to move to scan one ormore regions that are not covered by the other types of sensors of thefirst and second sets of sensors. In some embodiments, the one or moreregions that are not covered by the other types of sensors of the firstand second sets of sensors, are located within a range of 8 m or lessfrom the vehicle body. In some embodiments, the at least one of theplurality of ultrasonic sensors is configured to move relative to thevehicle body while the vehicle is in operation. In some embodiments, twoor more of the plurality of ultrasonic sensors are configured to moverelative to each other to cover a blind spot in proximity to thevehicle.

Aspects of the disclosure are directed to a vehicle comprising thesensing system as described previously herein. In some embodiments, thevehicle is a land-bound vehicle. In some embodiments, the vehiclecomprises space for one or more passengers. In some embodiments, thevehicle comprises one or more additional sensors configured to collectinformation about an environment in proximity to or around the vehicle.In some embodiments, the one or more additional sensors comprise one ormore of the following: GPS, infrared sensors, or wheel odometer. In someembodiments, the information from the one or more additional sensors iscombined with information from the first and second sets of sensors toaid in the autonomous operation of the vehicle.

Moreover, aspects of the disclosure provide a method for enablingautonomous operation of a vehicle, said method comprising: supporting,with aid of a vehicle body, a plurality of sensors comprising: (1) afirst set of sensors comprising two or more different types of sensorsoriented in a forward-facing direction and configured to detect two ormore regions in front of the vehicle, and (2) a second set of sensorscomprising one or more types of sensors oriented in a plurality ofdirections and configured to detect one or more regions in proximity toor surrounding the vehicle, wherein a range of each of the two or moreregions in front of the vehicle extends farther away from the vehiclecompared to a range of each of the one or more regions in proximity toor surrounding the vehicle; and collecting information from theplurality of sensors to aid in the autonomous operation of the vehicle.In some embodiments, the method further comprises: effecting a change inan orientation of at least one sensor from the first set of sensors inreal-time based on the vehicle's motion or predicted motion path.

It shall be understood that different aspects of the disclosure can beappreciated individually, collectively, or in combination with eachother. Various aspects of the disclosure described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of a vehicle may applyto and be used for any movable object, such as any vehicle.Additionally, the systems, devices, and methods disclosed herein in thecontext of ground motion (e.g., autonomous driving) may also be appliedin the context of other types of motion, such as movement in the air oron water, underwater motion, or motion in space.

Other objects and features of the present disclosure will becomeapparent by a review of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 shows an example of a vehicle with a sensing assembly, inaccordance with embodiments of the disclosure.

FIG. 2 shows an example of a sensing assembly on-board a vehicle, inaccordance with embodiments of the disclosure.

FIG. 3A shows examples of detectable ranges of various sensors that maydetect an environment around a vehicle, in accordance with embodimentsof the disclosure.

FIG. 3B shows additional examples of detectable ranges of varioussensors that may detect an environment around a vehicle, in accordancewith embodiments of the disclosure.

FIG. 4 provides an example of lidar units that may be part of a sensingassembly of a vehicle, in accordance with embodiments of the disclosure.

FIG. 5 shows an example of multiple groups of lidar units being arrangedon a vehicle, in accordance with an embodiment of the disclosure.

FIG. 6 shows an example of a vehicle with a plurality of groups of lidarunits, in accordance with embodiments of the disclosure.

FIG. 7 shows an example of a multi-lidar module including a long rangelidar unit, in accordance with embodiments of the disclosure.

FIG. 8 shows an example of multiple vision sensors being arranged on avehicle to provide a plurality of binocular cameras, in accordance withsome embodiments.

FIG. 9 shows an example of multiple binocular cameras being arranged ona vehicle for sensing various directions and ranges, in accordance withan embodiment of the disclosure.

FIG. 10 shows an example of a vehicle with a plurality of binocularcameras comprising various combinations of vision sensors, in accordancewith an embodiment of the disclosure.

FIG. 11 illustrates a binocular camera 134 for stereo vision, inaccordance with some embodiments.

FIG. 12 illustrates the transforming of relative coordinates of one ormore cameras to match the reference frame of the vehicle.

FIG. 13 illustrates a plurality of monocular cameras are supported ondifferent sides of a vehicle, in accordance with some embodiments.

FIG. 14 illustrates a vision sensing system comprising a plurality ofbinocular cameras and at least one monocular camera, in accordance withsome embodiments.

FIG. 15 illustrates a vision sensing system on a vehicle in accordancewith some embodiments.

FIG. 16 illustrates a radar system being arranged on a vehicle inaccordance with some embodiments.

FIG. 17 illustrates how one or more sensors may be configured to changeorientation based on a vehicle's motion or predicted motion inaccordance with some embodiments.

FIG. 18 provides an additional illustration of how one or more sensorsmay be configured to change based on a vehicle's motion or predictedmotion in accordance with embodiments.

FIG. 19 illustrates an ultrasonic sensing system being arranged on avehicle in accordance with some embodiments.

FIG. 20 illustrates a sensing system controller in communication with asensing assembly of a vehicle, in accordance with some embodiments.

FIG. 21 illustrates an automatic driving system 1900 comprising ahardware sensor module 1910, a sensing module 1930, and a navigation andposition module 1940, in accordance with some embodiments.

FIG. 22 illustrates the time synchronization of different types ofsensors in a sensor module, in accordance with some embodiments.

DETAILED DESCRIPTION

The present disclosure provides systems and methods for autonomouspiloting (driving) of a vehicle. The autonomous driving systemsdisclosed herein can (1) enable seamless environmental sensing in 360degrees, (2) provide redundant, reliable and stable environment sensingdata, and/or (3) effect an easy and quick sensor calibration and areal-time calibration verification. The disclosed systems can sense theenvironment in which the vehicle is being operated, and detect thepresence of stationary and moving obstacles. For example, the systemsdescribed herein can collect positional and/or motion information of thevehicle using a plurality of sensors, and control the vehicle (e.g.,with respect to position, velocity, and/or acceleration) to safelynavigate through various types of environments without collision withobstacles. In some cases, the autonomous driving systems can beconfigured to determine and select which sensors and/or sensing data toutilize under various driving and/or environmental type. In particular,the disclosed methods and systems can improve the operational safety ofa vehicle, and enable the vehicle to self-navigate through variousenvironments (e.g., indoors or outdoors, adverse weather conditions suchas rain, fog and snow, different types of obstacles located at variousdistances, unfamiliar terrain, high altitude or low altitude, etc.) in asafe manner. It shall be understood that different aspects of thedisclosure can be appreciated individually, collectively, or incombination with each other. Various aspects of the disclosure describedherein may be applied to any of the particular applications set forthbelow or for any other types of remotely controlled vehicles or movableobjects.

The embodiments disclosed herein can be applied to any suitable movableobject. The movable object can be configured to move within any suitableenvironment, such as on ground (e.g., a motor vehicle or a train), inair (e.g., a fixed-wing aircraft, a rotary-wing aircraft, or an aircrafthaving neither fixed wings nor rotary wings), in water (e.g., a ship ora submarine), in space (e.g., a spaceplane, a satellite, or a probe), orany combination of these environments. The movable object can be avehicle, such as an automobile.

FIG. 1 shows an example of a vehicle 100 with a sensing assembly 110, inaccordance with embodiments of the disclosure. The sensing assembly maybe on-board the vehicle. The vehicle may be capable of traveling withinan environment and collecting information about the environment with aidof the sensing assembly. The sensing assembly may aid in automateddriving by the vehicle. The automated driving system of the vehicle maycomprise the sensing assembly, which may comprise multiple sensors.

The vehicle 100 may be a sensing vehicle capable of sensing theenvironment in proximity to or around the vehicle. The vehicle may be aland-bound vehicle. The vehicle may travel over land. Alternatively orin addition, the vehicle may be capable of traveling on or in the water,underground in the air, and/or in space. The vehicle may be anautomobile. The vehicle may be a land-bound vehicle, watercraft,aircraft, and/or spacecraft. The vehicle may travel freely over asurface. The vehicle may travel freely within two dimensions. Thevehicle may primarily drive on one or more roads.

Optionally, the vehicle may be an unmanned vehicle. The vehicle may nothave a passenger or operator on-board the vehicle. The vehicle may ormay not have a space within which a passenger could ride. The vehiclemay or may not have space for cargo or objects to be carried by thevehicle. The vehicle may or may not have tools that may permit thevehicle to interact with the environment (e.g., collect samples, moveobjects). The vehicle may or may not have objects that may be emitted tobe dispersed to the environment (e.g., light, sound, liquids,pesticides). The vehicle may operate without requiring a human operator.

In some embodiments, the vehicle may permit one or more passengers toride on-board the vehicle. The vehicle may comprise a space for one ormore passengers to ride the vehicle. The vehicle may have an interiorcabin with space for one or more passengers. The vehicle may or may nothave an operator. For example, a vehicle may have a space for a driverof the vehicle. In some embodiments, the vehicle may be capable of beingdriven by a human operator. Alternatively or in addition, the vehiclemay be operated using an autonomous driving system.

In some embodiments, a vehicle may switch between a manual driving modeduring which a human driver would drive the vehicle, and an autonomousdriving mode during which an automated controller may generate signalsthat operate the vehicle without requiring intervention of the humandriver. In some embodiments, the vehicle may provide driver assistancewhere the driver may primarily manually drive the vehicle, but thevehicle may execute certain automated procedures or assist the driverwith performing certain procedures (e.g., lane changes, merging,parking, auto-braking). In some embodiments, the vehicle may have adefault operation mode. For example, the manual driving mode may be adefault operation mode, or an autonomous driving mode may be a defaultoperation mode.

A secondary operation mode may come into effect. For example, if themanual driving mode is the default operation mode, the autonomousdriving mode may be the secondary operation mode. If the autonomousdriving mode is the default operation mode, the manual driving mode maybe the secondary operation mode.

The secondary operation mode may come into effect as a result of userinput. For example, a user may start by driving in a manual drivingmode. Then, the user may provide an input to indicate a switch over toautonomous driving mode. The user may provide the input while driving.The user may provide the input while the vehicle is stationary. Inanother example, the user may start by having the car in an autonomousdriving mode. The user may provide an input that indicates the user istaking over manual control of the vehicle.

The secondary operation mode may come into effect as a result of asignal generated by one or more processors. The signal may be generatedby the one or more processors in response to a detected event. The eventmay be detected with aid of one or more sensors. For example, if theuser is in a manual driving mode and sensors detect large debris on theroad up-ahead, the system may automatically brake to avoid the debris,or swerve around it if the conditions are safe. The secondary operationmode may come into effect without requiring any human input.

The vehicle may be any type of vehicle. The vehicle may be a passengervehicle. Examples of vehicle types may include, but are not limited to,sedans, coupes, pickup trucks, hatchbacks, station wagons, mini-vans,vans, buses, crossovers, SUVs, convertibles, trucks, motorcycles, carts,flatbeds, semis, transport trucks, shuttles, all-terrain vehicles, orany other types of vehicles. The vehicle may be capable of transportingat least 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 16, 20, 30, 40, 50, 60, 70,or 100 people. The vehicle may have seats for any number of individuals,including the numbers listed herein.

The vehicle may comprise one or more propulsion units that may allow avehicle to traverse an environment. The propulsion units may compriseone or more wheels that may come into contact with an underlyingsurface. The propulsion units may comprise treads, rollers, propellers,rotor blades, and/or nozzles that may permit the vehicle to move. Thevehicle may comprise an engine or motor that may drive the vehicle.

The vehicle may operate with aid of a fuel. The vehicle may operateusing a gaseous fuel, liquid fuel, and/or solid fuel. The vehicle mayoperate using a diesel fuel. The vehicle may operate on a biofuel. Thevehicle may be an electric vehicle, or may be a hybrid electric vehicle.

The vehicle may comprise an on-board sensing assembly 110. The sensingassembly may be carried by the vehicle. The sensing assembly maycomprise one or more sensors capable of sensing the environment inproximity to or around the vehicle. External sensors may be capable ofcollecting data from the environment around the vehicle. The sensingassembly may comprise one or more sensors capable of sensing a conditionof the vehicle, or sensing a characteristic of a component on-board thevehicle. Internal sensors may be capable of collecting data regardingthe vehicle itself.

The sensing assembly may comprise a single sensor or multiple sensors.The sensing assembly may comprise a single type of sensor or multipletypes of sensors. Examples of types of sensors may include, but are notlimited to, location sensors (e.g., global positioning system (GPS)sensors, mobile device transmitters enabling location triangulation),vision sensors (e.g., imaging devices capable of detecting visible,infrared, or ultraviolet light, such as cameras), proximity sensors(e.g., ultrasonic sensors, lidar, time-of-movement cameras), inertialsensors (e.g., accelerometers, gyroscopes, inertial measurement units(IMUs)), altitude sensors, pressure sensors (e.g., barometers), audiosensors (e.g., microphones) or field sensors (e.g., magnetometers,electromagnetic sensors). Any suitable number and combination of sensorscan be used, such as one, two, three, four, five, or more sensors.Optionally, the data can be received from sensors of different types(e.g., two, three, four, five, or more types). Sensors of differenttypes may measure different types of signals or information (e.g.,position, orientation, velocity, acceleration, proximity, pressure,etc.) and/or utilize different types of measurement techniques to obtaindata. For instance, the sensors may include any suitable combination ofactive sensors (e.g., sensors that generate and measure energy fromtheir own source) and passive sensors (e.g., sensors that detectavailable energy). The sensing assembly may comprise a single sensor ofa particular sensor type or multiple sensors of the particular sensortype.

The sensors may have various detectable ranges. A detectable range ofthe sensor may include a region relative to the sensor where the sensormay collect data. The detectable range may include a distance rangeand/or a direction. For example, detectable range may include a maximumdistance and/or minimum distance that can be read by the sensor. Theminimum distance may be zero. The maximum distance may or may not beaffected by environmental conditions (e.g., temperature, particulates inthe air, precipitation, air pressure, noise, etc.). Direction mayinclude an angle range. For instance, a sensor may have an angular rangefield of view. Sensors may not be capable of collecting data outsidetheir detectable range. Areas outside the detectable range of aparticular sensor may be a blind spot of the sensor. Different sensorsmay have different detectable ranges or the same detectable range.

The sensors may be distributed anywhere on the vehicle. The vehicle maycomprise a vehicle body. The sensors may be distributed anywhere overthe vehicle body. The sensors may be within an interior of the vehiclebody, outside the vehicle body, or part of the vehicle body. The sensorsmay be distributed within a space defined by a vehicle chassis, outsidea space defined by a vehicle chassis or may be part of the vehiclechassis. The sensors may be within a vehicle housing, outside a vehiclehousing, or part of the vehicle housing. The sensors may be distributedon a top surface of a vehicle, bottom surface of a vehicle, frontsurface of a vehicle, rear surface of a vehicle, right side surface of avehicle or a left side surface of a vehicle. The sensors may bedistributed on an upper half of the vehicle, and/or lower half of thevehicle. The sensors may be distributed on a front half of the vehicleand/or a rear half of the vehicle. The sensors may be distributed arounda perimeter of the vehicle.

The sensors may be arranged to have various detectable ranges around thevehicle. The various detectable ranges may or may not overlap oneanother. The various detectable ranges may be sufficient to permit safeautonomous driving of the vehicle. The detectable ranges covered by thesensors may be sufficient to prevent accidents or unsafe driving of thevehicle when operating in an autonomous driving mode. The sensors may becapable of permitting the vehicle to drive autonomously on one or moreroads. The sensors may be capable of permitting the vehicle to drivesafely off-road.

A sufficient number and variety of types of sensors may be provided thatmay permit safe autonomous operation of the vehicle in differentenvironmental conditions. For instance the vehicle may be capable ofoperating safely in a wide range of temperatures (e.g., even in extremeheat or cold). The autonomous driving system of the vehicle may becapable of operating safely in conditions with poor visibility (e.g.,night time, heavy precipitation, fog, particulates in the air). Theautonomous driving system of the vehicle may be capable of operating inconditions with different atmospheric pressure or levels of moisture.The autonomous driving system of the vehicle may be capable of operatingin conditions with various types of precipitation (e.g., rain, hail,snow, sleet), various wind conditions, various road conditions, and/orvarious noise conditions. The number and/or types of sensors from thesensing assembly may be able to detect relevant information from theenvironment under the various types of conditions.

FIG. 2 shows an example of a sensing assembly 110 on-board a vehicle, inaccordance with embodiments of the disclosure. The sensing assembly maycomprise one or more types of sensors, such as lidar 120, cameras 130,radar 140, ultrasonic sensors 150, GPS 160 and/or odometers 170. The oneor more sensors carried by the vehicle may include, but are not limitedto location sensors (e.g., global positioning system (GPS) sensors,mobile device transmitters enabling location triangulation), visionsensors (e.g., imaging devices capable of detecting visible, infrared,or ultraviolet light, such as cameras), proximity sensors (e.g.,ultrasonic sensors, lidar, time-of-movement cameras), inertial sensors(e.g., accelerometers, gyroscopes, inertial measurement units (IMUs)),altitude sensors, pressure sensors (e.g., barometers), audio sensors(e.g., microphones) or field sensors (e.g., magnetometers,electromagnetic sensors). The sensors may be used to collect data of thesurrounding environment around the vehicle. Optionally, the sensors maybe used to collect data regarding the vehicle itself. Data from thesensors (e.g., of the surrounding environment and/or the vehicle itself)may be fused. Data from multiple types of sensors can be fused. Forinstance, data of the surrounding environment can be obtained insubstantially real-time by fusing information from multiple sensors.

The sensing assembly may comprise one or more lidar 120 units. The lidarunits may be single-channel lidars. The lidar units may be one or morescanning lidars. The lidar units may illuminate a target or detectablerange with laser light. The lidar units may be capable of detectingbackscattering. The light may comprise ultraviolet, visible, and/ornear-infrared light to image the surrounding environment. The lidarunits may be capable of detecting a wide range of materials. Forinstance, the lidar may detect metallic or non-metallic objects,precipitation, certain aerosols, clouds or molecules. In someembodiments, the lidar units may be operating at a high resolution. Anytype of lidar may be used, such as Rayleigh lidar, Mie lidar, Ramanlidar, Na/Fe/K lidar, etc. In some embodiments, the lidar units need notbe of a mechanical scanning type of lidar. For example, the lidar unitsmay include phase array lidars integrated on microchips. Advantages ofphase array lidars include lower cost, lower weight, smaller formfactor, and fewer mechanical components compared to existing scanninglidar systems. Phase array lidars are also more robust due to the lackof moving parts since the components are integrated onto microchips.

One or more cameras 130 may be part of the sensing assembly. The camerasmay collectively form a vision sensing system. Multiple cameras may beprovided. The cameras may be capable of capturing image data forenvironmental sensing. The cameras may be the same type of cameras ordifferent types of cameras. In some embodiments, the cameras may includestereo cameras. Optionally, the cameras may include one or moremonocular cameras. In some embodiments, combinations of stereo camerasand monocular cameras may be provided. The cameras may include black andwhite cameras. In some embodiments, the cameras may include colorcameras. Any description herein of cameras may apply to any type ofvision sensors, and may be referred to interchangeably as imagingdevices of which examples are described below.

An imaging device may be a physical imaging device. An imaging devicecan be configured to detect electromagnetic radiation (e.g., visible,infrared, and/or ultraviolet light) and generate image data based on thedetected electromagnetic radiation. An imaging device may include acharge-coupled device (CCD) sensor or a complementarymetal-oxide-semiconductor (CMOS) sensor that generates electricalsignals in response to wavelengths of light. The resultant electricalsignals can be processed to produce image data. The image data generatedby an imaging device can include one or more images, which may be staticimages (e.g., photographs), dynamic images (e.g., video), or suitablecombinations thereof. The image data can be polychromatic (e.g., RGB,CMYK, HSV) or monochromatic (e.g., grayscale, black-and-white, sepia).The imaging device may include a lens configured to direct light onto animage sensor.

The imaging device can be a camera. A camera can be a movie or videocamera that captures dynamic image data (e.g., video). A camera can be astill camera that captures static images (e.g., photographs). A cameramay capture both dynamic image data and static images. A camera mayswitch between capturing dynamic image data and static images. Althoughcertain embodiments provided herein are described in the context ofcameras, it shall be understood that the present disclosure can beapplied to any suitable imaging device, and any description hereinrelating to cameras can also be applied to any suitable imaging device,and any description herein relating to cameras can also be applied toother types of imaging devices. A camera can be used to generate 2Dimages of a 3D scene (e.g., an environment, one or more objects, etc.).The images generated by the camera can represent the projection of the3D scene onto a 2D image plane. Accordingly, each point in the 2D imagecorresponds to a 3D spatial coordinate in the scene. The camera maycomprise optical elements (e.g., lens, mirrors, filters, etc). Thecamera may capture color images, greyscale image, infrared images, andthe like. The camera may be a thermal imaging device when it isconfigured to capture infrared images.

In some embodiments, the cameras 130 may include multiple imagingdevices, or an imaging device with multiple lenses and/or image sensors.The cameras may be capable of taking multiple images substantiallysimultaneously. The multiple images may aid in the creation of a 3Dscene, a 3D virtual environment, a 3D map, or a 3D model. For instance,a right image and a left image may be taken and used for stereo-mapping.A depth map may be calculated from a calibrated binocular image. Anynumber of images (e.g., 2 or more, 3 or more, 4 or more, 5 or more, 6 ormore, 7 or more, 8 or more, 9 or more) may be taken simultaneously toaid in the creation of a 3D scene/virtual environment/model, and/or fordepth mapping. The images may be directed in substantially the samedirection or may be directed in slightly different directions. In someinstances, data from other sensors (e.g., ultrasonic data, LIDAR data,data from any other sensors as described elsewhere herein, or data fromexternal devices) may aid in the creation of a 2D or 3D image or map.

An imaging device may capture an image or a sequence of images at aspecific image resolution. In some embodiments, the image resolution maybe defined by the number of pixels in an image. In some embodiments, theimage resolution may be greater than or equal to about 352×420 pixels,480×320 pixels, 720×480 pixels, 1280×720 pixels, 1440×1080 pixels,1920×1080 pixels, 2048×1080 pixels, 3840×2160 pixels, 4096×2160 pixels,7680×4320 pixels, or 15360×8640 pixels. In some embodiments, the cameramay be a 4K camera or a camera with a higher resolution.

An imaging device may capture a sequence of images at a specific capturerate. In some embodiments, the sequence of images may be capturedstandard video frame rates such as about 24p, 25p, 30p, 48p, 50p, 60p,72p, 90p, 100p, 120p, 300p, 50i, or 60i. In some embodiments, thesequence of images may be captured at a rate less than or equal to aboutone image every 0.0001 seconds, 0.0002 seconds, 0.0005 seconds, 0.001seconds, 0.002 seconds, 0.005 seconds, 0.01 seconds, 0.02 seconds, 0.05seconds. 0.1 seconds, 0.2 seconds, 0.5 seconds, 1 second, 2 seconds, 5seconds, or 10 seconds. In some embodiments, the capture rate may changedepending on user input and/or external conditions (e.g. rain, snow,wind, unobvious surface texture of environment).

An imaging device may have adjustable parameters. Under differingparameters, different images may be captured by the imaging device whilesubject to identical external conditions (e.g., location, lighting). Theadjustable parameter may comprise exposure (e.g., exposure time, shutterspeed, aperture, film speed), gain, gamma, area of interest,binning/subsampling, pixel clock, offset, triggering, ISO, etc.Parameters related to exposure may control the amount of light thatreaches an image sensor in the imaging device. For example, shutterspeed may control the amount of time light reaches an image sensor andaperture may control the amount of light that reaches the image sensorin a given time. Parameters related to gain may control theamplification of a signal from the optical sensor. ISO may control thelevel of sensitivity of the camera to available light. Parameterscontrolling for exposure and gain may be collectively considered and bereferred to herein as EXPO.

In some alternative embodiments, an imaging device may extend beyond aphysical imaging device. For example, an imaging device may include anytechnique that is capable of capturing and/or generating images or videoframes. In some embodiments, the imaging device may refer to analgorithm that is capable of processing images obtained from anotherphysical device.

The sensing system may comprise one or more radar systems 140. The radarsystems may use radio waves to detect objects in the environment aroundthe vehicle. The radar system may comprise a transmitter producingelectromagnetic waves in the radio or microwaves domain, and atransmitting antenna. The radar system may comprise a receiving antenna(which may be the same as the transmitting antenna). The radar systemmay comprise a receiver and a processor that may determine properties ofany detected objects. Radio waves may be sent from the transmitter to bereflected off of any objects in the detectable range, and return to thereceiver.

In some embodiments, the radar may be an extremely high frequency (EHF)radar, such as a millimeter wave radar. In some alternative embodiments,the radar may be at a super high frequency band, or the far infraredband. The radar may have a band of radio frequencies in theelectromagnetic spectrum from 30 to 300 gigaHertz. The radio waves mayhave wavelengths from ten to one millimeters.

One or more ultrasonic sensors 150 may be part of the sensing system ofthe vehicle. The ultrasonic sensors may comprise ultrasonic transmittersthat convert electrical signals into ultrasound. The ultrasound signalsmay be emitted and reflected. The reflected ultrasound may be convertedby receivers into electrical signals. The ultrasound transmitters andreceivers may or may not be part of the same transceivers.

The sensing assembly may comprise one or more global positioning system(GPS) sensors 160. The GPS sensor may be used to detect a geolocation ofthe vehicle. Any description herein of a GPS sensor may apply to anytype of global navigation satellite system (GNSS) sensor. The GPS sensormay communicate with one or more satellites to provide autonomousgeo-spatial positioning. The GPS sensor may comprise a small electronicreceiver that determines its location (e.g., longitude, latitude, and/oraltitude/elevation) using time signals transmitted along a line of sightfrom the satellites. The signals may be transmitted via radio. In someembodiments, the GPS sensor may be capable of detecting the geolocationof the vehicle to a high degree of precision (e.g., within a fewmeters).

In some embodiments, a wheel odometer 170 may be provided as part of thesensing assembly. The wheel odometer may be used to calculate a distancetraveled by the vehicle. If a signal by a GPS sensor is lost, the datafrom the wheel odometer may be used to detect how far a vehicle hastraveled and estimate a location of the vehicle based on the last knownlocation.

A set of sensor modules may be built with the various sensors to cover acertain detection area and range. The sensing assembly may comprise aplurality of sensor modules that may relate to the various types ofsensors as provided. For example, the sensing assembly may comprise oneor more multi-lidar modules, one or more multi-camera modules, one ormore multi-radar modules, one or more ultrasonic sensor modules, and/orone or more wheel odometer and GPS modules. A module may comprise one ormore sensors. For instance a multi-lidar module may comprise multiplelidar sensors.

Data from each of the sensors may be collected. In some instances, datafrom sensors in a module may be collected and/or aggregated. The datafrom the sensors in a module may be analyzed separately or together.Data from multiple modules of a vehicle may be collected and/oraggregated. The data from the modules may be analyzed separately ortogether. For instance, data from multiple modules with different typesof sensors may be fused together. Sensor fusion may also include datafrom multiple types of sensors to be used together to aid in operationof the vehicle.

In some embodiments, sensing results are generated by combining sensordata obtained by multiple sensors using any known sensor fusiontechniques, which can include algorithms based on a Kalman filter, anextended Kalman filter (EKF), an unscented Kalman filter (UKF), aparticle filter (PF), or suitable combinations thereof. For instance,sensor fusion can be used to combine sensing data obtained by differentsensor types, including as GPS sensors, inertial sensors, visionsensors, lidar, ultrasonic sensors, and so on. As another example,sensor fusion can be used to combine different types of sensing data,such as absolute measurement data (e.g., data provided relative to aglobal coordinate system such as GPS data) and relative measurement data(e.g., data provided relative to a local coordinate system such asvision sensing data, lidar data, or ultrasonic sensing data). Sensorfusion can be used to compensate for limitations or inaccuraciesassociated with individual sensor types, thereby improving the accuracyand reliability of the final sensing result.

The data obtained by the sensing assembly can provide various types ofenvironmental information. For example, the sensor data may beindicative of an environment type, such as an indoor environment,outdoor environment, low altitude environment, or high altitudeenvironment. The sensor data may also provide information regardingcurrent environmental conditions, including weather (e.g., clear, rainy,snowing), visibility conditions, wind speed, time of day, and so on.Furthermore, the environmental information collected by the sensors mayinclude information regarding objects in the environment such asobstacles described herein. Obstacle information may include informationregarding the number, density, geometry, and/or spatial disposition ofobstacles in the environment.

The data collected from the various sensors of the sensing assembly maybe obtained and analyzed with aid of one or more processors. The one ormore processors may aggregate the data from the various sensors, whichmay include multiple sensor types. The one or more processors maygenerate one or more instructions that may affect operation of thevehicle based on the analyzed data from the sensors. The analysis andgeneration of the instructions may occur substantially in real-time. Forinstance, the instructions that may affect operation of the vehicle maybe generated within about 1 minute, 30 seconds, 20 seconds, 15 seconds,10 seconds, 7 seconds, 5 seconds, 3 seconds, 2 seconds, 1 second, 0.5seconds, 0.3 seconds, 0.1 seconds, 0.05 seconds, 0.01 seconds, 0.005seconds, or 0.001 seconds of the data being collected by the sensors.

In some embodiments, the one or more processors can process the datafrom the sensing assembly, and select which set of sensors and/or datato utilize for sensing the surrounding environment. The processors canbe located onboard or remote from the vehicle. The data collected by theselected sensors can be provided to the automated driving systemdescribed herein. The automated driving system can be configured tocontrol one or more propulsion units of the vehicle to effect motionbased on the sensing data. For example, the sensing data provided by theselected sensors can be used to control the spatial disposition,velocity, and/or orientation of the vehicle (e.g., using a suitableprocessing unit and/or control module, as described elsewhere herein).Additionally, the sensors can be used to provide data regarding theenvironment proximate to or surrounding the vehicle, such as weatherconditions, proximity to potential obstacles, location of geographicalfeatures, location of manmade structures, and the like, as describedabove.

The instructions affecting operation of the vehicle may includeinstructions for driving the vehicle. The instructions may be used toenable autonomous driving of the vehicle. Autonomous driving of thevehicle may enable the vehicle to drive safely to a destination.Autonomous driving of the vehicle may include causing the vehicle toremain on the roadway. Autonomous driving of the vehicle may includedetection of an available lane. Autonomous driving of the vehicle mayinclude detection of other vehicles and pedestrians. Autonomous drivingof the vehicle may include collision avoidance, with one or more othervehicles, pedestrians, and/or objects.

The vehicle may optionally comprise one or more communication units. Thecommunication unit may permit the vehicle to communicate with one ormore external devices. The external device may be one or more datacenters that may collect and/or aggregate information the vehicle and/orother vehicles. The one or more data centers may be provided on one ormore external devices, such as one or more servers, personal computers,mobile devices, and/or via a cloud computing or peer-to-peerinfrastructure. In some embodiments, the external device may be anothervehicle.

The communication unit may permit wireless communication between thesensing vehicle and one or more external devices. The communication unitmay permit one-way communication (e.g., from the sensing vehicle to theexternal device, or from the external device to the sensing vehicle),and/or two-way communications (e.g., between the sensing vehicle and oneor more external devices). The communication unit may have a limiteddistance or range. The communication unit may be capable of long-rangecommunications. The communication unit may engage in point-to-pointcommunications. The communication unit may be broadcasting information.

In one example, the communication unit may comprise one or moretransceivers. The communication unit may comprise a transmitter and/or areceiver. The communication unit may be configured for any type ofwireless communication as described elsewhere herein. The communicationunit may comprise one or more antennas that may aid in thecommunications. The communication unit may or may not include acommunication dish. The communication unit may be directional (e.g.,operate strongest in a specified direction) or may operate substantiallyuniformly across all directions.

A communication unit 102 may be in communication with one or moresensors. The communication unit may receive data collected by the one ormore sensors. In some embodiments, data collected by one or more sensorsmay be transmitted using the communication unit. The data transmitted bythe communication unit may optionally be raw data collected by the oneor more sensors. Alternatively or in addition, the data transmitted bythe communication unit may be pre-processed on-board the vehicle. Insome embodiments, a vehicle may have one or more on-board processorsthat may perform one or more pre-processing steps on the data collectedby the sensors, prior to transmission of data to the communication unit.The pre-processing may or may not include formatting of the data into adesired form.

The pre-processing may or may not include analysis of the sensor datawith respect to the sensing vehicle and/or with respect to an inertialreference frame (e.g., the environment). For instance, thepre-processing may or may not include determination of positionalinformation relating to one or more surrounding vehicles or vehicleitself. The positional information may be with respect to the vehicle orwith respect to the inertial reference frame (e.g., geo-spatialcoordinates). For instance, the vehicle may be able to determinelocation and/or movement information for the vehicle or one or moresurrounding vehicles. The vehicle may be able to detect location and/ormovement of other objects within the environment, such as pedestrians,animals, structures, curbs, walkways, trees, bushes, bumpers, sidewalks,or so forth.

The communication unit may be positioned anywhere on or in the vehicle.The communication unit may be provided within a volume contained by oneor more body panels of the vehicle. The communication unit may beprovided within a volume within a vehicle chassis. The communicationunit may be external to a housing or body of the vehicle.

The vehicle may comprise one or more on-board processors. The one ormore processors may form an on-board computer or controller. Forinstance, the vehicle may comprise an electronic control unit (ECU). TheECU may provide instructions for one or more activities of the vehicle,which may include, but are not limited to, propulsion, steering,braking, fuel regulation, battery level regulation, temperature,communications, sensing, or any other operations. The one or moreprocessors may be or may comprise a central processing unit (CPU),graphics processing unit (GPU), field-programmable gate array (FPGA),digital signal processor (DSP) and so forth.

FIG. 3A shows examples of detectable ranges of various sensors that maydetect an periphery environment in proximity to or around a vehicle, inaccordance with embodiments of the disclosure. A vehicle 100 maycomprise a sensing assembly 110 which may comprise one, two or moredifferent sensors that may be capable of collecting information for anenvironment around the vehicle. For example, sensors from a first sensortype may collectively have a first detectable range 101 a, sensors froma second sensor type may collectively have a second detectable range 101b, sensors from a third sensor type may collectively have a thirddetectable range 101 c, sensors from a fourth sensor type maycollectively have a fourth detectable range 101 d, sensors from a fifthsensor type may collectively have a fifth detectable range 101 e,sensors from a sixth sensor type may collectively have a sixthdetectable range 101 f, and/or sensors from a seventh sensor type maycollectively have a seventh detectable range 101 g. Any of the sensortypes provided herein may include lidar, cameras, radar, ultrasonicsensors, GPS, odometers, inertial sensors, or any other sensors, such asthose described elsewhere herein.

Although various detectable ranges are illustrated with various shapes,it may be understood that the detectable ranges may have any shape. Forexample, the detectable range may have a substantially circular shape.The vehicle may be located at the center of the circle or another partof the circle. The detectable range may have a substantially ellipsoidalor oval shape. The detectable range may have a substantially sector orwedge shape. The detectable range may have a substantially triangularshape, quadrilateral shape (e.g., rectangular shape, square shape,diamond shape, trapezoidal shape), pentagonal shape, hexagonal shape,octagonal shape, or any other shape. Any of the shapes described hereinmay represent a cross-section of the detectable range. In someembodiments, the shapes may be a lateral cross-sectional shape, or avertical cross-sectional shape. The detectable range may form aspherical, semi-spherical, conical, cylindrical, prismatic, toroidal, orany other type of shape. In some embodiments, the detectable range maycomprise a combination or a plurality of any of the shapes described, tocollectively form a new shape. The detectable range may be formed of asingle continuous shape or multiple discontinuous shapes.

The detectable range may collectively reach around at least 360 degreessurrounding the vehicle. In some instances, the detectable range may beat least about 15 degrees, 30 degrees, 45 degrees, 60 degrees, 75degrees, 90 degrees, 120 degrees, 150 degrees, 180 degrees, 210 degrees,240 degrees, 270 degrees, 300 degrees, or 330 degrees around thevehicle. The detectable range may have angular values less than any ofthe values described herein, or falling within a range between any twoof the values described herein. The angle ranges may be providedrelative to a lateral direction around the vehicle, or verticaldirection around the vehicle. In some embodiments, the detectable rangesmay be evenly distributed around the vehicle. In some embodiments, someof the detectable ranges for various sensors may be distributed more andextending away from the front of the vehicle, the rear of the vehicle,the top of the vehicle, a right side of the vehicle, a left side of thevehicle, or a bottom side of the vehicle, or any combination of thesesides of the vehicle. The distribution of the sensors may depend on thetypes of sensors. The distribution of the sensors may depend on the typeof objects or environmental conditions that the sensors are used todetect.

A first sensor type may comprise lidar units. One or more lidar unitsmay be provided on-board the vehicle. The one or more lidar units maycollectively have a first detectable range 101 a. The first detectablerange may have a radius of d1. The radius d1 may represent the maximumrange of the first detectable range. In some embodiments, d1 may beabout 80 m. In some embodiments, the detectable range may have a maximumvalue about 20 m, 30 m, 40 m, 50 m, 60 m, 70 m, 75 m, 80 m, 85 m, 90 m,100 m, 110 m, 120 m, 130 m, 150 m, 175 m, 200 m, or 250 m. In someembodiments, the detectable range by the lidar units may encompass 360degrees around the vehicle. In some embodiments, the collectivedetectable range by the lidar units may have a substantially circularshape around the vehicle. In some embodiments, the collective detectablerange of by the lidar units may comprise a plurality of cones or wedgeshapes around the vehicle.

A second sensor type may comprise stereo cameras. One or more stereocameras may be provided on-board the vehicle. The one or more stereocameras may collectively have a second detectable range 101 b. Thesecond detectable range may have a radius of d2. The radius d2 mayrepresent the maximum range of the second detectable range. In someembodiments, d2 may be about 60-100 m. In some embodiments, thedetectable range may have a maximum value about 5 m, 10 m, 20 m, 30 m,40 m, 50 m, 55 m, 60 m, 65 m, 70 m, 75 m, 80 m, 85 m, 90 m, 95 m, 100 m,105 m, 110 m, 120 m, 130 m, 150 m, 175 m, 200 m, or 250 m. In someembodiments, the detectable range by encompass 360 degrees around thevehicle. In some embodiments, the collective detectable range by thestereo cameras may have a substantially circular shape around thevehicle. In some embodiments, the collective detectable range of thestereo cameras may comprise a plurality of cones or wedge shapes aroundthe vehicle.

A third sensor type may comprise ultrasonic sensors. One or moreultrasonic sensors may be provided on-board the vehicle. The one or moreultrasonic sensors may collectively have a third detectable range 101 c.The third detectable range may have a radius of d3. The radius d3 mayrepresent the maximum range of the third detectable range. In someembodiments, d3 may be about 8 m. In some embodiments, the detectablerange may have a maximum value about 0.1 m, 0.5 m, 1 m, 1.5 m, 2 m, 2.5m, 3 m, 4 m, 5 m, 6 m, 7 m, 8 m, 9 m, 10 m, 11 m, 12 m, 13 m, 15 m, 20m, 30 m, or 50 m. In some embodiments, the detectable range by encompass360 degrees around the vehicle. In some embodiments, the collectivedetectable range by the ultrasonic sensors may have a substantiallycircular shape around the vehicle. In some embodiments, the collectivedetectable range of by the ultrasonic sensors may comprise a pluralityof cones or wedge shapes around the vehicle.

A fourth sensor type may comprise radar, such as millimeter wave radar.One or more radar systems may be provided on-board the vehicle. The oneor more radar systems may collectively have a fourth detectable range101 d. The fourth detectable range may have a distance range of d4. Thedistance range d4 may represent the maximum range of the fourthdetectable range. In some embodiments, d4 may be about 180 m. In someembodiments, the detectable range may have a maximum value about 20 m,30 m, 50 m, 75 m, 100 m, 120 m, 150 m, 160 m, 170 m, 180 m, 190 m, 200m, 220 m, 250 m, 300 m, or 500 m. In some embodiments, the detectablerange by encompass a front region of the vehicle. In some embodiments,the collective detectable range by the radar systems may have asubstantially conical shape or wedge shape.

A fifth sensor type may comprise long range lidar. The long range lidarmay have a narrow field of view (FOV) but is not limited thereto.Different fields of view for the long lidar ranging from narrow to widecan be configured depending on the optical configuration of the lidar.In some embodiments, one or more long range lidar units may be providedon-board the vehicle. The one or more long range lidar units maycollectively have a fifth detectable range 101 e. The fifth detectablerange may have a distance range of d5. The distance range d5 mayrepresent the maximum range of the fourth detectable range. In someembodiments, d5 may be about 200 m. In some embodiments, the detectablerange may have a maximum value about 20 m, 30 m, 50 m, 75 m, 100 m, 120m, 150 m, 170 m, 180 m, 190 m, 200 m, 210 m, 220 m, 230 m, 250 m, 300 m,or 500 m. In some embodiments, the detectable range by encompass a frontregion of the vehicle. In some embodiments, the collective detectablerange by the long range lidar unit may have a substantially conicalshape or wedge shape.

A sixth sensor type may comprise a camera, such as a monocular camera.One or more monocular cameras may be provided on-board the vehicle. Theone or more monocular cameras may collectively have a sixth detectablerange 101 f. The sixth detectable range may have a distance range of d6.The distance range d6 may represent the maximum range of the sixthdetectable range. In some embodiments, d6 may be about 230 m. In someembodiments, the detectable range may have a maximum value about 20 m,30 m, 50 m, 75 m, 100 m, 120 m, 150 m, 160 m, 170 m, 180 m, 200 m, 210m, 220 m, 225 m, 230 m, 240 m, 250 m, 270 m, 300 m, or 500 m. In someembodiments, the detectable range by encompass a front region of thevehicle. In some embodiments, the collective detectable range by themonocular camera may have a substantially conical shape or wedge shape.

A seventh sensor type may comprise a second radar, such as millimeterwave radar, a second monocular camera, an additional long range lidarunit, or any other type of sensor. The sensor may be a rear-facingsensor. The one or more rear facing sensors may collectively have aseventh detectable range 101 g. The seventh detectable range may have adistance range of d7. The distance range d7 may represent the maximumrange of the fourth detectable range. The distance value may be any ofthe distance values described elsewhere herein. In some embodiments, thedetectable range by encompass a rear region of the vehicle. In someembodiments, the collective detectable range by the rear-facing sensormay have a substantially conical shape or wedge shape.

Detection ranges of a multi-sensor system are shown. Data from varioussensors can be fused before feeding to a detection algorithm. Asillustrated, different sensors and/or sensor types may have differentdetectable ranges that may collectively encompass the vehicle. Somesensors may have different distance ranges than others. For instance,some sensors may be able to reach greater distances than others. Somesensors may encompass different angular ranges than others. Some sensorsmay encompass wider ranges around the vehicle, while some sensors mayhave more narrow angular ranges. In some instances, some of the sensorswith a greater distance range may focus on the front and/or rear of thevehicle. This may be useful for detecting objects of interest as thevehicle drives.

An automatic driving system of the vehicle may be able to monitor thesurrounding environment of the vehicle with aid of the one or moresensors. The one or more sensors may aid in automated driving in one ormore ways. For instance, one or more of the sensors may be used todetect movement of remote objects to provide an early warning. One ormore sensors may be used to detect objects nearby, including those thatmay be in the blind spot of a vehicle operator, passenger, or othersensors.

Any of the sensors provided herein may be fixedly coupled to thevehicle. Any of the sensors provided herein may be rigidly coupled tothe vehicle. Any of the sensors may remain stationary relative to thevehicle.

Any of the sensors provided herein may move relative to the vehicle. Anyof the sensors provided herein may rotate relative to the vehicle. Theone or more sensors may rotate about one axis, two axes, or three axes,relative to the vehicle. Any of the sensors provided herein may movetranslationally relative to the vehicle. For instance, the one or moresensors may slide relative to the vehicle. The one or more sensors mayslide along a first axis, second axis, and/or third axis.

In some embodiments, the one or more sensors may rotate relative to thevehicle with aid of a carrier. The carrier may comprise a gimbal. Thegimbal may be a single-axis gimbal or multi-axis gimbal. The gimbal maybe a one-axis, two-axis, or three-axis gimbal. The gimbal may permitrotation about the yaw axis, the pitch axis, and/or the roll axis. Insome embodiments, the gimbal may permit rotation about the yaw axisonly, the pitch axis only, or both the yaw and the pitch axis. Thegimbal may comprise a frame assembly comprising one or more framecomponents that may move relative to one another to permit rotation ofthe sensor. In some embodiments, a first frame component may be coupledto the sensor. The sensor may be fixed relative to the frame componentor may rotate relative to the frame component about a first axis. Thefirst frame component may be optionally supported by a second framecomponent. The first frame component may rotate relative to the secondframe component. In some embodiments, the first frame component rotatesrelative to the second frame component about a second axis. The secondaxis may be different from the first axis. The second frame componentmay be supported by a third frame component. The second frame componentmay rotate about a third axis relative to the third frame component. Thethird axis may be different from the first and second axis. The gimbalmay comprise a motor assembly comprising one or more motors that maydrive movement of the frame components. For example, a first motor maydrive movement of a sensor relative to the first frame component. Asecond motor may drive movement of a first frame component relative to asecond frame component. A third motor may drive movement of a secondframe component relative to a third frame component.

The carrier may comprise one or more sensors that may be useful fordetermining orientation of the sensor relative to the vehicle and/or aninertial reference frame. The carrier may be used to control rotation ofthe sensors in response to instructions. The instructions may begenerated with aid of one or more processors on-board the vehicle, or atan external device or cloud computing infrastructure external to thevehicle. In some embodiments, the instructions may be generated based onmovement of the vehicle, predicted movement of the vehicle,environmental conditions, and/or external objects.

The one or more sensors may move translationally relative to the vehiclewith aid of one or more actuators. In some embodiments, a guide or trackmay be utilized that may allow the sensors to move translationally. Thesensors may move along the guide or track. The one or more actuators maydrive movement of the sensor along the guide or track. The guide ortrack may be substantially straight or may have a bend or curve. In someembodiments, multiple guide or tracks may intersect and the movement ofthe sensors may be transition between any of the guide or tracks at theintersections.

FIG. 3B shows additional examples of detectable ranges of varioussensors that may detect an environment around a vehicle, in accordancewith embodiments of the disclosure. Any variation of the variousdetectable ranges for different types of sensors as described elsewhereherein, may apply.

In one example, a vehicle 100 may comprise a sensing system with a radarhaving a first detectable range 103 a (e.g., 180 meters or more), sonarhaving a second detectable range 103 b (e.g., 7 meters or more), 1080pcameras having a third detectable range 103 c (e.g., 100 meters ormore), 4k camera having a fourth detectable range 103 d (e.g., 200meters or more), and/or lidar units having a fifth detectable range. Insome embodiments, the various detectable ranges may be sectors ofcircles, circles, cones, or any other shape or combination thereof.

In some embodiments, a sensing system may be provided, to aid inautonomous operation of the vehicle. The sensing system may comprise afirst set of sensors and a second set of sensors. The first set ofsensors may comprise two or more different types of sensors. The firstset of sensors may be oriented in a forward-facing direction andconfigured to detect two or more regions in front of the vehicle. Theregions may or may not overlap with one another. In some embodiments, afirst detectable region may lie completely within a second detectableregion. Optionally, a portion of the first detectable region lies withinthe second detectable region, and a portion of the first detectableregion lies outside the second detectable region. The first and seconddetectable regions may have different ranges. For example, a range of asecond detectable region may be greater than a range of the firstdetectable region. An area or volume of the first detectable region maybe determined by a scan angle of a first sensor type and an area orvolume of a second detectable region may be determined by a scan angleof a second sensor type. The scan angles of the first and second sensortypes may be different or the same. For instance, the scan angle of thefirst sensor type may be greater than a scan angle of the second sensortype. The detection range of the first sensor type may be less than orequal to the detection range of the second sensor type. Different sensortypes may be detecting the two or more regions. The second set ofsensors may comprise one or more types of sensors oriented in aplurality of directions and configured to detect one or more regionssurrounding or in proximity to the vehicle. The second set of sensorsmay be configured to collectively detect an area at least 180, 270, or360 degrees around the vehicle. The range of the first set of sensorsmay extend farther away from the vehicle compared to the range from thesecond set of sensors. In some embodiments, the first set of sensors andthe second set of sensors may share some sensors that are of the sametype. In some embodiments, the first set of sensors and the second setof sensors may share some common sensors (i.e. one or more sensors maybe commonly utilized by both the first and second sets of sensors).

In some embodiments, the first set of sensors may comprise a monocularcamera, a long range lidar unit, and/or a millimeter-wavelength radarunit. In some embodiments, one or more sensors may be orientedbackwards. The backwards facing sensors may comprise a monocular camera,a long range lidar unit, and/or a millimeter-wavelength radar unit. Insome embodiments, more forward-facing sensors may be provided thanbackward-facing sensors. In some embodiments, one or more sensors facingforward may operate with a higher resolution or precision than one ormore of the backward facing sensors. For example, a forward-facingmonocular camera may have a higher imaging resolution than abackward-facing monocular camera. For example, the forward-facingmonocular camera may have a 4K imaging resolution while abackward-facing monocular camera may have a 1080p imaging resolution.

The various sensors described herein may be suitable for use within anenvironment that the vehicle is traversing. In some embodiments, somesensor types may be more suited than others for operating in variousenvironmental conditions. For instance, a first sensor type may be moresuitable for use in a first type of environment and a second sensor typemay be more suitable for use in a second type of environment. The firstand second types of environments may have at least one differingenvironmental condition relative to one another. For instance, the firstand second environmental type may have different lighting conditions.The first and second environment types may have different objectdensities, different types of objects, and/or different sizes ofobjects. The first and second types of environments may have differentvisibility ranges. The first and second types of environments may havedifferent background noises or vibrations. The first and second types ofenvironment may have different types or degrees of particulates in theair. The first and second types of environment may experience differenttemperatures. The first and second types of environment may experiencedifferent precipitation. For example, factors, such as rain, snow, hail,sleet, fog, smog, dust, wind, smoke, cloudiness, time of day,temperature, may affect the type of environment.

Sensors may be selectively turned on or off, or used for variousenvironmental conditions. For example, a first sensor type may operatewell in a first environment type and poorly in a second environmenttype. A second sensor type may operate well in a second environment typeand poorly in a first environment type. A first sensor type may beconfigured to actively detect a region when the vehicle is operating inthe first type of environment. The second sensor type may be configuredto be passive or inactive when the vehicle is operating in the firsttype of environment. A second sensor type may be configured to activelydetect a region when the vehicle is operating in the second type ofenvironment. The first sensor type may be configured to be passive orinactive when the vehicle is operating in the second type ofenvironment. In some embodiments, the first sensor type, the secondsensor type, or both may be configured to collect data as the vehicle ismoving through the first and second types of environments. In oneexample, the data from the first type of sensor is processed and thedata from the second type of sensor is not processed when the vehiclemoving is through the first environment type. The data from the secondtype of sensor may be provided and the data from the first type ofsensor may not be processed when the vehicle is moving through thesecond environment type. Data from one or more types of sensors may ormay not be processed or analyzed depending on the environment typewithin which the vehicle is operating and the suitability of thosesensor types for that environment type. In some embodiments, somesensors may be capable of operating in multiple environment types. Suchsensors may be collecting data and/or processing/analyzing data whilethe vehicle is operating in the multiple environment types.

In some embodiments, one or more sensors on-board a vehicle may detectthe type of environment that the vehicle is operating in. In otherembodiments, data from outside the vehicle may be used to determine theenvironment that the vehicle is operating in. For instance, data from anexternal sensor off-board the vehicle may be used to collect informationabout the environment. Online data sources, such as weather reports, maybe used to determine environmental conditions. In some embodiments,external data sources may be used in combination with a map of anenvironment that the vehicle is navigating to determine the environmenttype. The environment type may be changed over time, when the vehicle ismoving or stationary. The environment type information may be updatedperiodically.

In response to detecting the environment type, an assessment may be madeof the suitability of one or more sensor types for operating within thedetected environment type. In some embodiments, when one or more sensorsare determined to be suitable to operate within the detected environmenttype, they may be used to actively collect data. The data collected bythe sensors may be processed and/or analyzed. When one or more sensorsare determined to be unsuitable for operation within the detectedenvironment type, they may be inactive or used to passively collectdata. In some embodiments, they may collect data, but the data may notbe processed and/or analyzed. Or the data that is processed and/oranalyzed may be discounted or weighted less than data from sensors thatare suitable for the environment.

FIG. 4 provides an example of lidar units that may be part of a sensingassembly of a vehicle, in accordance with embodiments of the disclosure.Part A shows an example of a single lidar unit 120. Part B shows anexample of a plurality of lidars that may be commonly supported. Part Cshows an example of a detectable range of a plurality of lidars.

Part A shows an example of a single lidar unit 120. A vehicle sensingassembly may comprise a single lidar unit or multiple lidar units. Alidar unit may be any type of lidar. In some embodiments, the lidar unitmay be a single-channel lidar. The lidar unit may be a scanning lidar.Optionally, one or more lidar units on the vehicle may not be part of amulti-channel monolithic lidar unit. In some embodiments, none of thelidar units on the vehicle are part of a multi-channel monolithic lidarunit. In some embodiments, any type of lidar unit may be used with anynumber of channels (e.g., 1, 2, 4, 8, 16, 24, 32, or 64 channels). Thelidar unit may be a multi-channel lidar unit. The lidar unit may operateat any sampling frequency. For example, the lidar unit may emit at least5K, 10K, 20K, 30K, 50K, 75K, 100K, 200K, or 500K pulses per second.

The lidar unit may have any detectable range. In one example, the lidarunit may have a substantially conical detectable range, with the pointof the cone at the location of the lidar unit. The lidar unit may have aprimary direction, which may intersect the center of the cone. The conemay have any field of view (FOV). In some embodiments, the lidar unitmay have a FOV of at least 15, 30, 45, 55, 57, 60, 75, 90, 120, 150,180, 210, 240, 270, 300, 330, or 360 degrees. The lidar unit may have aFOV with an angular range less than any of the values provided herein orfalling within a range between any two of the values provided herein.The lidar unit may be scanning anywhere within the detectable range. Thelidar unit can control a direction of the emitted laser at a certainrate to cover a detectable range, such as a conical detectable range asdescribed.

One or more lidar units may be supported by a vehicle. The lidar unitsmay be distributed in any manner on the vehicle. In some embodiments, aplurality of lidar units may be distributed on the vehicle to detect a360 degree region around the vehicle. The plurality of lidar units maybe arranged so that a set of lidar units is supported by a commonsupport.

Part B shows an example of a plurality of lidar units 122-1, 122-2,122-3 on a common support 102. A set 122 of lidar units may comprise twoor more lidar units that are on a common support. A set of lidar unitsmay comprise one or more, two or more, three or more, four or more, fiveor more, six or more, seven or more, or eight or more lidar units on acommon support. The lidar units in the set may be clustered together.

The lidar units may be arranged to be pointing at different directionsfrom one another. The primary direction of each of the lidar unitswithin the set may be different from one another. The primary directionsof each of the lidar units may be non-parallel. The primary directionsof each of the lidar units of the set may or may not intersect at acommon point. The lidar units may be pointing toward one another.Alternatively, the lidar units may be pointing away from one another.

The lidar units may have a fixed position relative to one another. Thecommon support may provide a support structure that may keep the lidarunits at fixed positions relative to one another. Each lidar unit of agroup may be fixedly attached to its respective support structure.During operation of the vehicle, the lidar units may remain at fixedpositions relative to one another. During operation of the vehicle, thelidar units may remain at fixed positions relative to the vehicle body.During operation of the vehicle, the lidar units may remain at fixedpositions relative to the support structure. The lidar units may remainfixed with aid of a fixture device configured to rigidly affix the lidarunits. During operation of the vehicle, the lidar units within a groupmay be configured to move relative to one another. Movement of thevehicle may cause less than a 5 degree, 3 degree, 2 degree, 1 degree,0.5 degree or 0.1 degree variance in the angles relative to one anotherand/or relative to the environment. Such movement of less than thedegrees provided may constitute the lidar units being substantiallyfixed. The support structure may be formed from a substantially rigidmaterial. In some alternative embodiments, the lidar units may moverelative to one another. During operation of the vehicle, the lidarunits may move relative to the vehicle body. The support structure maycomprise one or more hinges, ball joints, tracks, slides, grooves, orother mechanisms that may allow the lidar units to move relative to oneanother. The support structure may comprise one or more actuator thatmay cause the lidar units to move relative to one another. In someembodiments, the lidar units may be supported by a carrier on thesupport structure. The carrier may be gimbal as described elsewhereherein. The carrier may comprise a one-axis gimbal, two-axis gimbal, orthree-axis gimbal. The lidar may rotate about a yaw, pitch, and/or rollaxis relative to the support structure. In some embodiments, at somemoment in time, the carrier may hold the lidar units at fixed positionsrelative to one another, the support structure, and/or the vehicle body.In some embodiments, the carrier may permit movement about one, two, ormore degrees of freedom relative to the support structure, vehicle, orinertial reference frame, to maintain a fixed disposition between lidarunits within the same set. The lidar units may rotate about the sameamount in the same direction. In some instances, the fixed dispositionmay be maintained with aid of one or more linkages. The linkages maycomprise serial or parallel linkages. The linkages may be multi-barlinkages. The fixed disposition may be maintained with aid of akinematic coupling. The fixed disposition may be maintained bymechanically coupling the lidar units in a rigid manner. The dispositionof the lidar units may be controlled in real-time. The disposition ofthe lidar units may be controlled during operation of the vehicle.

The lidar units may be held within a recess or sleeve of the commonsupport. The lidar units may be attached with aid of brackets, or othertypes of fasteners, to the common support. The lidar units may becompletely or partially embedded in the common support. The lidar unitson a common support may be located close to one another. In someembodiments, there may be a distance of less than 30 cm, 20 cm, 15 cm,10 cm, 7 cm, 5 cm, 3 cm, 2 cm, 1 cm, 0.5 cm, or 0.1 cm between adjacentlidar units within the same set. The lidar units may be supported by thesupport structure. The weight of the lidar units may be borne by thesupport structure.

Part C shows an example of a set 122 of lidar units 122-1, 122-2, 122-3on a common support 102. Each of the lidar units may comprise adetectable range. The detectable range for each of the lidar unitsrelative to the respective lidar units may be the same. For example,each lidar unit may have the same detectable distance and/or FOV angle.In one example, each lidar unit has a 57 degree FOV, or any other valueas described elsewhere herein. The detectable range for each of thelidar units relative to an inertial reference frame (e.g., theenvironment) may be different from one another. The difference may bedue to the placement of the lidar units relative to one another.

In some embodiments, the lidar units may be arranged on the commonsupport so that they are in the same plane. The lidar units may be onsubstantially the same lateral plane. The lidar units may be at the sameelevation above ground. The lidar units may be at the same height on thevehicle. The lidar units may be arranged so that the detectable rangesare directed primarily laterally. The lidar units may be substantiallyoriented horizontally. There may or may not be a vertical component tothe direction of the lidar units. A vertical component of the directionof the lidar units may be less than or equal to about 15 degrees, 10degrees, 5 degrees, 3 degrees, or 1 degree.

The lidar units within the same set may be arranged to all have the samevertical degree of orientation. For instance, all of the lidar units maybe arranged with zero degrees of vertical orientation. In anotherexample, all of the lidar units within the set may be angled slightlyupwards, or may be angled slightly downwards. Alternatively, lidar unitswithin the same set may have slightly different vertical orientations.For example, a first lidar unit within a set may be angled slightlyupwards, while the other two lidar units are angled slightly downwardsor straight horizontally. In another example, two of the lidar units maybe angled slightly upwards while a third lidar unit may be angledslightly downwards or straight horizontally. In some embodiments, lidarunits within the same set or between different sets may have slightlydifferent vertical orientations or substantially different verticalorientations. The variations in vertical orientations allow the lidarunits to adequately detect different types objects of various heights(e.g., children who may be below a certain height and not easilydetected, small animals such as pets, bicycles, motorcycles, trucks suchas 18-wheelers, trucks with tailgates, etc.).

The detectable ranges of the lidar units within a set may or may notoverlap with one another. The lidar units may be arranged so that theirFOVs may or may not overlap. In some embodiments, their FOVs may overlapby less than 15 degrees, 10 degrees, 5 degrees, 3 degrees, 2 degrees, or1 degree. In some instances, their FOVs may overlap more than any of thevalues provided herein. The FOVs may overlap within a range between anytwo values provided herein. In some embodiments, the detectable rangemay not overlap at all.

The detectable ranges of the plurality of lidar units of a set maycollectively form a detectable range for the set. For instance, if eachof the lidar units have a detectable range with a 57 degree FOV, and thelidar units are arranged so that the detectable ranges are right next toeach other to form a single continuous range without substantialoverlap, then the collective detectable range may have about a 171degree FOV. In some embodiments, the lidar units within the same set maybe shifted about 30 degrees, 40 degrees, 45 degrees, 50 degrees, 55degrees, 57 degrees, 60 degrees, 65 degrees, 70 degrees, 80 degrees, 90degrees or any other value. The lidar units within the same set may beshifted by at least any of these degree values, no more than any ofthese degree values or within a range falling between any of thesedegree values. The lidar units of the set may have detectable rangesthat are arranged to form a single continuous detectable range for theset. Alternatively, there may be gaps that may cause multiplediscontinuous detectable ranges for the set. A collective field of viewof a lidar unit of a set may be adjustable in real-time by changing aposition of at least one lidar unit within the set. This adjustment mayoccur while a vehicle is in operation. Such adjustments may occur inreal-time.

In some embodiments, overlap may be provided between the detectableranges of the lidar units of the same set. The lidar units may bearranged in a manner to increase overlap between adjacent detectableranges of the lidar units. Increased overlap may include overlap of atleast 1 degree, 3 degrees, 5 degrees, 10 degrees, 15 degrees, 30degrees, 45 degrees, 60 degrees, 75 degrees, or 90 degrees.

In some embodiments the set of lidar units may have a collective fieldof view of at least 30 degrees, 45 degrees, 60 degrees, 90 degrees, 120degrees, 135 degrees, 150 degrees, 160 degrees, 171 degrees, 180degrees, 210 degrees, 240 degrees, 270 degrees, 300 degrees, 330degrees, or 360 degrees.

FIG. 5 shows an example of multiple groups of lidar units being arrangedon a vehicle, in accordance with an embodiment of the disclosure. Avehicle 100 may comprise a plurality of sets of lidar units that arearranged to form a multi-lidar module. Each set of lidar units may be asubset of the lidar units of the vehicle. Each subset of lidar units maycomprise two or more lidar units. Each lidar unit of each subset may notoverlap with lidar units in other subsets. In some embodiments, no twosubsets of lidar units may share lidar units with one another. A lidarunit may belong to a single lidar unit.

Any number of lidar units may be arranged on the vehicle body. In someembodiments, less than or equal to about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10,12, 15, 20, 25, 30, 35, 40, 50, or 100 lidar units are used to cover adesired detectable region around the vehicle. Optionally, more than anyof the numbers of lidar units provided herein may be used. In someembodiments, the number of lidar units used to cover a desireddetectable range may fall within a range between any two of thesenumbers.

A driving assembly may comprise a plurality of lidar units that may beconfigured to be supported by a vehicle body. The lidar units may becollectively configured to detect a periphery region in proximity to oraround the vehicle body. The lidar units may be useful for aiding inautonomous driving upon coupling the driving assembly to the vehiclebody. Each of the plurality of lidar units may have any detectable rangeangle, such as those described elsewhere herein (e.g., less than 180degrees). The lidar units may comprise a first subset of lidar unitscomprising at least two lidar units having a fixed disposition relativeto one another and a second subset of lidar units comprising at leasttwo lidar units having a fixed disposition relative to one another. Thefirst and second subset of lidar units may be supported on differentareas of the vehicle body and may be configured to work in concert todetect a periphery region in proximity to or around the vehicle body ora portion thereof. The subset of lidar units may be groupings orclusters of lidar units.

Any number of groupings of lidar units may be provided. For instance,one or more, two or more, three or more, four or more, five or more, sixor more, seven or more, eight or more, nine or more, or ten or moregroupings of lidar units may be provided. Each grouping of lidar unitsmay have a common support. The groupings of lidar units may be arrangedso that multiple clusters of lidar units are arranged on a vehicle.

Each grouping of lidar units may have the same characteristics. Forexample, each grouping of lidar units may have the same detectable rangerelative to the grouping. Each grouping of lidar units may have the samemaximum distance or FOV angle. Alternatively, one or more of thegrouping of lidar units may have different characteristics. One or moregrouping of lidar units may have a different detectable range. One ormore groupings of lidar units may have a different maximum distance orFOV angle.

In one example, four groups of lidar units 122, 124, 126, 128 may beprovided on a vehicle. Any description herein may apply to any number ofgroups of lidar units on the vehicle. The groups of lidar units may bearranged so that overlap is provided between the groups of the lidarunits. Overlap may or may not be provided between any two adjacentgroups of lidar units. In some embodiments, overlap between two groupsof lidar units, such as adjacent groups of lidar units may be at least 5degrees, 15 degrees, 30 degrees, 45 degrees, 60 degrees, 70 degrees, 80degrees, 90 degrees, 105 degrees, 120 degrees, 150 degrees, 180 degrees,210 degrees, 240 degrees, 270 degrees, or 360 degrees. Optionally, theoverlap between two groups of lidar units may be less than any of thevalues provided herein or within a range falling between any two of thelidar units provided herein. The degree of overlap between the groups oflidar units may be sufficient for sensor calibration and/or real timeerror detection of sensor calibration.

Each of the groups of sensors may have a support for the lidar units ofthe group. Each group of lidar units may comprise multiple lidar units.For example, a first support structure may support one, two, three ormore lidar units, a second support structure may support one, two, threeor more lidar units, a third support structure may support one, two,three or more lidar units, and/or a fourth support structure may supportone, two, three or more lidar units. Each support structure may beseparate from one another. The support structure may be substantiallystationary relative to one another on a vehicle. Alternatively, thesupport structures may move relative to one another. In someembodiments, operation of the vehicle may cause motion that may causethe support structures to move slightly relative to one another. Datafrom different support structures may be calibrated relative to oneanother.

In some embodiments, the support structures may move relative to thevehicle body. The support structure may rotate or slide relative to thevehicle body. The support structure may be rigidly affixed to thevehicle body. The support structure may be attached to the vehicle bodyvia one or more dampeners (e.g., shock absorbing mechanisms). Examplesof dampeners may comprise springs, elastic materials (e.g., rubbers,foams), pneumatic dampeners, or hydraulic dampeners. The supportstructure may be attached to the vehicle with aid of a carrier, such asa gimbal as described elsewhere herein.

In some embodiments, a first support structure may move relative to asecond support structure. A first group of lidar units supported by thefirst support structure may move relative to a second group of lidarunits supported by the second support structure. The groups of lidarunits may move to focus on a predefined portion of a region around thevehicle. The groups of lidar units may move while the vehicle is inoperation. The groups of lidar units may be configured to move with aidof one more carriers. The one or more carriers may effect rotation ofthe support structures, or may effect rotation of individual lidarunits. In some embodiments, the predefined portion of the region mayhave a different object density than the rest of the region around thevehicle. The predefined region may have a higher object density than therest of the region. Object density may be based on a number of objectswithin a volume of space. Object density may be based on a percentage ofa volume of space that is taken up by one or more objects. Objectdensity may be based on a percentage of volume of space that is nottaken up by one or more objects having a continuous volume that meets aminimum threshold.

The groups of lidar units may move relative to one another to adjust anoverlap of field of view between the groups of lidar units. A firstgroup of lidar units may move relative to a second group of lidar unitsto adjust an overlap of field of view between the first and secondgroups of lidar units. In some embodiments, the overlap of the field ofview may be adjusted in real-time. The overlap of the field of view maybe adjusted during operation of the vehicle. In some embodiments, theoverlap of the field of view may be adjusted to compensate for blindspots while the vehicle is in operation. In some embodiments, theadjustment may be made based on activity of the vehicle, such as achange or predicted change in direction. In some embodiments, theadjustment may be made based on conditions around the vehicle, such astraffic coming from a particular side of the vehicle.

The collective field of view of the lidar units may be adjustable bycausing groups of lidar units to move relative to one another. Thecollective field of view of lidar units may be adjustable by changing aposition of a first group of lidar units relative to a second group oflidar units. A collective field of view of a plurality of groups oflidar units may be inversely proportional to a collective detectionrange of the groups of lidar units. In some embodiments, an increase ofthe collective field of view of the groups of lidar units may cause thecollective detection range of the groups of lidar units to decrease. Thecollective field of view and detection range may be adjustable inreal-time while the vehicle is in operation. This may depend on a widthof the collective field of view and/or distance of the collectivedetection range that is being selected.

In some embodiments, an autonomous driving assembly of a vehicle maycomprise a plurality of lidar units configured to be supported by thevehicle body, wherein the lidar units may be collectively configured todetect a 360 degree region around the vehicle body, or any other degreeregion as provided elsewhere herein, to aid in the autonomous drivingupon coupling said driving assembly to the vehicle body. The pluralityof lidar units may comprise a first subset of lidar units comprising atleast two lidar units supported by a first support structure and asecond subset of lidar units comprising at least two lidar unitssupported by a second support structure. The first and second supportstructures may be separate from one another. Optionally, the pluralityof lidar units may comprise a third subset of lidar units comprising atleast two lidar units supported by a third support structure. The thirdsupport structure may be separate from the first and second supportstructure. The plurality of lidar units may comprise a fourth subset oflidar units comprising at least two lidar units supported by a fourthsupport structure. The fourth support structure may be separate from thefirst, second, and/or third support structure. In some embodiments, thesubsets of lidar units may be oriented facing diagonally outward fromdifferent corners of the vehicle. For example, the first subset of lidarunits may be oriented facing outward along a diagonal extending from thefirst corner of the vehicle. Similarly, the second subset of lidar unitsmay be oriented facing outward along a diagonal extending from thesecond corner of the vehicle, the third subset of lidar units may beoriented facing outward along a diagonal extending from the third cornerof the vehicle, and the fourth subset of lidar units may be orientedfacing outward along a diagonal extending from the fourth corner of thevehicle. The diagonals extending from different corners of the vehiclemay or may not be on a same horizontal plane. In some cases, two or moreof the diagonals may lie on a common axis. Alternatively, all of thediagonals may lie on different axes. The diagonals may extend, forexample at about 45 degrees from each respective corner of the vehicle.The diagonals may extend at an acute angle or oblique angle. In someembodiments, the first subset of lidar units may be primarily orientedat 45 degrees, the second subset of lidar units may be primarilyoriented at 135 degrees, the third subset of lidar units may beprimarily oriented at 225 degrees, and/or the fourth subset of lidarunits may be primarily oriented at 315 degrees, facing away from thevehicle along diagonals extending respectively from the first, second,third and fourth corners of the vehicle.

As described above, each subset of lidar units may be supported by itsrespective support structure. In some embodiments, two or more subsetsof lidar units may be supported by a same or single support structure.For example, the first and second subsets of lidar units may besupported by one support structure, and the third and fourth subsets oflidar units may be supported by another support structure. The first andsecond subsets of lidar units may be rigidly attached to one supportstructure. Similarly, the third and fourth subsets of lidar units may berigidly attached to the other support structure. The aforementionedsupport structures may be located on different sides of the vehicle, andcan be fixed or movable relative to each other.

In some embodiments, all of the subsets of lidar units may be supportedon a same support structure. For example, the first, second, third andfourth subsets of lidar units may be supported on a single supportstructure that is attached or coupled to the vehicle. The single supportstructure may be provided as a frame, plate, truss, or sheet, and may beformed of a rigid material (e.g., metal or fiber composite). The supportstructure may be rigidly coupled to the chassis of the vehicle. Eachsubset of lidar units may be attached to a different end or corner of asingle support structure. In some embodiments, the single supportstructure may have a rectangular shape, and the first, second, third andfourth subsets of lidar units may be attached to the respective fourcorners of the rectangular-shaped support structure. Each subset oflidar units may be rigidly attached or movably attached to eachrespective corner of the support structure. In some embodiments, all ofthe subsets of lidar units may be rigidly attached to the same (orsingle) support structure, such that all of the subsets of lidar unitshave a fixed spatial disposition relative to one another. The fixedspatial disposition of the subsets of lidar units may be maintained atall times, for example even during motion of the vehicle. In some cases,the single support structure need not have a rectangular shape, and canbe formed in a variety of regular shapes (e.g., triangular, pentagonal,hexagonal, etc.) or irregular shapes. In those cases, a subset of lidarunit may be attached to each corner. For example, when the supportstructure is formed having a hexagonal shape, six different subsets oflidar units may be respectively attached to the six corners of thehexagonal-shaped support structure. In some embodiments, the singlesupport structure may be formed having a shape that substantiallyfollows the contour or profile of the vehicle body.

In some further embodiments, one or more support structures need not beused to support the subsets of lidar units. In some cases, the omissionof support structures can be advantageous and help reduce the weight andcost of the vehicle. One or more subsets of lidar units may be attacheddirectly to the vehicle body without any intervening support structure.In some embodiments, all of the subsets of lidar units may be attacheddirectly to the vehicle body. For example, a first subset of lidar unitsmay be directly attached to a first corner of the vehicle body, a secondsubset of lidar units may be directly attached to a second corner of thevehicle body, a third subset of lidar units may be directly attached toa third corner of the vehicle body, and a fourth subset of lidar unitsmay be directly attached to a fourth corner of the vehicle body. Asdescribed elsewhere herein, the first subset of lidar units may beprimarily oriented facing outward along a first diagonal from the firstcorner of the vehicle, the second subset of lidar units may be primarilyoriented facing outward along a second diagonal from the second cornerof the vehicle, the third subset of lidar units may be primarilyoriented facing outward along a third diagonal from a third corner ofthe vehicle, and the fourth subset of lidar units may be primarilyoriented facing outward along a fourth diagonal from a fourth corner ofthe vehicle.

The groups of lidar units may be arranged to be pointing at differentdirections from one another. A primary direction of a group of lidarunits may be at a center of a FOV of the collective lidar units. Theprimary direction of each of the groups of lidar units may be differentfrom one another. The primary directions of each of the groups of lidarunits may be non-parallel. The primary directions of each of the lidarunits of the set may or may not intersect at a common point. The groupsof lidar units may be pointing away from one another. Alternatively, thegroups of lidar units may be pointing toward one another.

In some embodiments, the groups of lidar units may be arranged so thattheir respective common supports are in the same plane. The groups oflidar units may be on substantially the same lateral plane. The supportstructures may be on substantially the same lateral plane.Alternatively, the groups of lidar units may be arranged such that theirrespective common supports are on different planes. Two or more planesmay be parallel to one another. Alternatively, two or more planes neednot be parallel, and may intersect one another. In some embodiments, thegroups of lidar units may be arranged such that some of the commonsupports are on a same plane and the remaining common supports may be ondifferent planes. The groups of lidar units may be at the same elevationabove ground. The support structures may be at the same elevation aboveground. The groups of lidar units may be at the same height on thevehicle. The support structures may be at the same height on thevehicle. The groups of lidar units may be arranged so that thedetectable ranges are directed primarily laterally. The groups of lidarunits may be substantially oriented horizontally. There may or may notbe a vertical component to the primary direction of the groups of lidarunits. A vertical component of the direction of the groups of lidarunits may be less than or equal to about 15 degrees, 10 degrees, 5degrees, 3 degrees, or 1 degree.

Each group of lidar units may have identical arrangements. For instance,each group of lidar units may have the same number of lidar units and/ortypes of lidar units. Each group of lidar units may have the samesupport structure. Each group of lidar units may have the samedetectable ranges relative to the lidar units and/or the supportstructure. Alternatively, two or more of the groups of lidar units mayhave different characteristics from one another (e.g., different number,different type of lidar units, different support structure, differentdetectable ranges, etc.).

The groups of lidar units may be arranged on a vehicle so that they have90 degree offsets relative to one another. For instance, a first groupmay have a primary direction that is about 90 degrees relative to aprimary direction of a second group. The second group may have a primarydirection that is about 90 degrees relative to a third group. The thirdgroup may have a primary direction that is about 90 degrees relative toa fourth group. The fourth group may have a primary direction that isabout 90 degrees relative to the first group. Depending on the group oflidar units, they may have different degrees of offset relative to oneanother. For instance, if there are N groups of lidar units, the groupsof lidar units may have a 360/N degree offset relative to one another.The groups of lidar units may be evenly spaced or angled relative to oneanother. Alternatively they need not be evenly spaced or angled relativeto one another.

In some embodiments, the groups of lidar units may be positioned at ornear corners of the vehicle. The groups of lidar units may be positionedat or near top corners of the vehicle (e.g., the roof corners of thevehicle). The groups of lidar units may be positioned at or near the farcorners of the vehicle (e.g., the main body of the vehicle). The groupsof lidar units may be positioned at or near the front bumper cornersand/or the rear bumper corners. The groups of lidar units may bepositioned at or near the front hood corners, and/or the rear trunkcorners. Corners of the vehicle may be provided where two sides comestogether. Corners of the vehicle may be provided where two sides havingdifferent orientations intersect with one another. In some embodiments,corners of the vehicle may be provided where three sides havingdifferent orientations intersect one another. For example, an upperright front corner may be provided where a front surface, right surface,and top surface intersect.

The groups of lidar units may be directed to be about 45 degrees (or amultiple thereof) offset a direction of motion of the vehicle. The groupof lidar units may be directed to about 45 degrees offset a length ofthe vehicle (e.g., running from the front F to rear R of the vehicle).For example, a first group 122 of lidars may be offset by about 315degrees from an axis running along the length of the vehicle, a secondgroup 124 may be offset by about 45 degrees from an axis running alongthe length of the vehicle, a third group 126 may be offset by about 225degrees from an axis running along the length of the vehicle, and/or afourth group 128 may be offset by about 135 degrees from an axis runningalong the length of the vehicle. The groups of lidar units may bedirected toward the corners of the vehicle.

FIG. 6 shows an example of a vehicle 100 with a plurality of groups 122,124, 126, 128 of lidar units, in accordance with embodiments of thedisclosure. The groups of lidar units may be located anywhere on thevehicle. The lidar units may be supported by a vehicle body. The weightof the lidar units may be borne by the vehicle body. The lidar units mayor may not directly contact the vehicle body. The lidar units may movewith the vehicle body. The lidar units may be affixed relative to thevehicle body.

In some embodiments, the lidar units may be located at or near a ceilingof the vehicle. For example, the lidar units may be located on a toproof of the vehicle facing away from the vehicle. The lidar units may belocated at or near a top of a cabin of the vehicle. The lidar units maybe located on, outside, or within body panels of the vehicle. The lidarunits may be located at top corners of the vehicle. The lidar units maybe located at top corners of the cabin of the vehicle. The lidar unitsmay be located at the top corners of the ceiling of the vehicle. Forinstance, the lidar units may be located at a top right front corner,top right rear corner, top rear left corner, and/or a top left frontcorner of the passenger cabin of the vehicle. The lidar units may belocated at a position above or in line with a head of an operator orpassenger of the vehicle. The groups of lidar units may be facing awayfrom one another. The groups of lidar units may be facing outwards intothe environment around the vehicle. The lidar units may or may not belocated at or near the highest point of the vehicle. The lidar units maybe located at or near the top 50%, 40%, 30%, 25%, 20%, 15%, 10%, 5%, 3%,or 1% of the vehicle.

Alternatively or in addition, the lidar units may be located at or neara bottom portion of the vehicle. The lidar units may be located at ornear the bottom 50%, 40%, 30%, 25%, 20%, 15%, 10%, 5%, 3%, or 1% of thevehicle. The lidar units may be locate at or near the corners of theentirety of the vehicle body. The lidar units may be located at or nearthe bumpers of the vehicle. The lidar units may be located at or nearthe front panels of the vehicle, the rear panels of the vehicle, and/orthe side panels of the vehicle. The lidar units may be at a height nearor in line with the front or rear lights of the vehicle. The lidar unitsmay be located at or near a hood and/or trunk of the vehicle. The lidarunits may be located at or near the corners of a hood and/or trunk ofthe vehicle.

The groups of lidar units may be located at one or more corners of thevehicle. For instance, a first group of lidar units may be located at afirst corner of the vehicle while a second group of lidar units may belocated at a second corner of the vehicle. The first and second cornersmay be located on the same side of the vehicle or different sides of thevehicle. The first and second corners may be located on opposite sidesof the vehicle or adjacent sides of the vehicle. A corner may be locatedat, or defined by the intersection of two laterally adjacent sides ofthe vehicle. There may be a third group of lidar units located at athird corner of the vehicle and a fourth group of lidar units located ata fourth corner of the vehicle. The third and fourth corners may belocated opposite to the first and second corners of the vehicle. Thefirst and second groups of lidar units may be located on a same side ordifferent sides of the vehicle. The first and second groups of lidarunits may be supported by the same body panel of the vehicle ordifferent body panels of the vehicle. The first group of lidar units maybe located on a first side of the vehicle and a second group of lidarunits may be located on a second side of the vehicle. The first andsecond sides may be adjacent to one another or may be opposite oneanother. A third and fourth group of lidar units may be located on athird side and a fourth side of the vehicle respectively. In someembodiments, at least two of the first, second, third, and fourth sidesmay be opposite one another. At least two of the first, second, third,and fourth sides may be adjacent to one another.

The groups of lidar units may be oriented so that they substantiallyface outward from the corners of the vehicle. Two or more groups oflidar units may substantially face outward from corners at the front Fof the vehicle and/or two or more groups of lidar units maysubstantially face outward from corners at the rear R of the vehicle.Alternatively or in addition, the groups of lidar units may be orientedso they face the primary directions of the vehicle. For instance, one ormore groups of lidar units may substantially face the front of thevehicle, one or more groups of lidar units may substantially face therear of the vehicle, one or more groups of lidar units may substantiallyface a right side of the vehicle, and/or one or more groups of lidarunits may substantially face a left side of the vehicle. The groups oflidar units may be facing away from one another. The groups of lidarunits may be facing away from a center of the vehicle. In someembodiments, at least four groups of lidar units may be facing away fromone another. In some instances, at least two of the four groups of lidarunits may be oriented at directions orthogonal to one another.Optionally, at least two of the four groups of lidar units may beoriented at directions parallel to one another. Alternatively or inaddition, at least two of the four groups of lidar units may be orientedat directions oblique to one another.

The one or more groups of lidar units may be substantially facinghorizontally. The one or more groups of lidar units may substantially befacing laterally outwards. The one or more groups of lidar units may ormay not have a vertical component to their primary direction. The one ormore groups of lidar units may have a vertical component of less than orequal to about 30 degrees, 15 degrees, 10 degrees, 5 degrees, 3 degrees,or 1 degree, or any other degree value provided elsewhere herein. Thegroups of lidar units may be angled downwards, upwards, or straighthorizontally.

Each group of lidar units may comprise a plurality of lidar units on acommon support. The support structure may be located on a body panel ofthe vehicle, part of the body panel of the vehicle, or within a bodypanel of the vehicle. The support structures may be located on, or maybe part of a vehicle chassis. The support structures may or may not beremovable from the rest of the vehicle. The support structure may bepermanently affixed or integral of the vehicle. The support structuresmay have a fixed position relative to the rest of the vehicle. Thesupport structures may be fixed relative to the vehicle panels and/orchassis. The support structures may or may not be movable relative tothe vehicle panels and/or chassis. The support structures may or may notcomprise an actuator that move the support structure relative to thevehicle panels and/or chassis.

Clustering the lidar units as indicated may allow a wide range to bedetected around the vehicle, using simple lidar units. For instance,single channel lidars may be used to detect at least 180, 270, 360 orany other degree value as described elsewhere herein, around thevehicle. Lidar units with less than 55 degrees, 57 degrees, or 60degrees FOV may be used to detect regions around the vehicle. In someembodiments, fewer than 30, 25, 20, 18, 16, 15, 14, 13, 12, 10, 8, 6, or4 lidar units may be used to detect a desired angular range while havinga FOV with any of the values described elsewhere herein. In someembodiments, fewer than 10, 8, 6, 5, 4, 3, or 2 groups of lidar unitsmay be used. The lidar units may be arranged so that each group (e.g.,cluster) of lidar units may be overlap with an adjacent group of lidarunits. The degree of overlap may be sufficient so that even if there isan error in a particular lidar unit, the error can be compensated by theoverlap.

Clustering the lidar units may be advantageous since the relativepositions between lidar units within the same group may be fixed andknown. The lidar units within the same group may be supported by thesame support structure, which may keep the positions between the lidarunits known and fixed. Thus, during operation of the vehicle,calibration between the lidar units within the same group may not berequired. In some embodiments, calibration between the lidar unitswithin the same group may occur upon manufacture of the vehicle.Calibration between the lidar units within the same group may occur uponpowering up the vehicle and/or initialization of the vehicle.Calibration of the lidar units may occur in response to a detectedevent. Calibration may occur prior to utilization of the groups of lidarunits. The calibration may be intrinsic calibration. Calibration of thelidar units within the same group may not be required during regularoperation of the vehicle.

Calibration may occur between different groups of lidar units. Since thelidar units are clustered into groups, there may be less calibrationrequired than if the lidar units are each separately supported. In someembodiments, calibration between different groups of lidar units mayoccur upon manufacture of the vehicle. Calibration between differentgroups of lidar units may occur upon powering up the vehicle and/orinitialization of the vehicle. Calibration different groups of lidarunits may occur in response to a detected event. Calibration differentgroups of lidar units may be required during regular operation of thevehicle. Calibration may occur on a regular basis. However, clusteringof lidar units may advantageously require less calibration than if thelidar units are each separately supported. For instance, if 12 lidarunits are separately supported, they may each need to be calibratedtoward one another. If the 12 lidar units are clustered into fourgroups, then only calibration between the four groups may be required.

Single channel lidar units may be cost-effective and easily accessiblecompared to multi-channel lidar units. In some embodiments,alternatively or in addition to the groups of lidar units describedherein, a multi-channel lidar unit may be used to detect the environmentaround the vehicle. For example, a 64-channel lidar unit, such as aVelodyne 64-channel lidar unit, may be used. In some embodiments, acombination of single channel lidar units and multi-channel lidar unitsmay be utilized, either individually or collectively, on a vehicle forenvironmental sensing.

FIG. 7 shows an example of a multi-lidar module including a long rangelidar unit 129, in accordance with embodiments of the disclosure. Insome embodiments, the long range lidar unit may have a narrow field ofview (FOV) but is not limited thereto. Different fields of view for thelong lidar ranging from narrow to wide can be configured depending onthe optical configuration of the lidar. A multi-lidar module maycomprise one or more lidar units with a first range, and may compriseone or more lidar units with a second range. The second range may have agreater maximum distance than the first range.

For example, the multi-lidar module may comprise a plurality of groupsof lidar units having a first range. The first range may have anydistance value as provided elsewhere herein. In some embodiments, thefirst range may have a maximum distance of about 80. The multi-lidarmodule may comprise one or more additional lidar units having a secondrange. The second range may have any distance value as providedelsewhere herein. In some instances, the second range may have a maximumdistance of about 200 m. In some embodiments, the second range may havea greater maximum distance than the first range. The second range mayhave a maximum distance that is at least 10%, 20%, 30%, 40%, 50%, 75%,100%, 150%, 200%, 300%, or 500% greater than the maximum distance of thefirst range. The first range and second range may have the same FOVangle, or may have different FOV angles. A lidar unit with the firstrange may have a FOV angle than is greater than, less than, or equal tothe FOV angle of a lidar unit with the second range. In someembodiments, the long range lidar unit may have a narrower FOV than theother types of lidar units. The second range may have a narrower FOVangle than the first range.

As illustrated, the long range lidar unit 129 may be forward-facing. Thelong range lidar unit may have a primary direction to the front, andfacing out from the front side of the vehicle. The long range lidar unitmay or may not deviate from its primary direction. The long range lidarunit may be substantially directed laterally. The long range lidar unitmay or may not have a vertical component to its primary direction. Thelong range lidar unit may have a vertical direction within 30 degrees,15 degrees, 10 degrees, 5 degrees, 3 degrees, 1 degree, or any otherdegree value, of the horizontal direction. The long range lidar unit maybe facing any direction. Alternatively or in addition to facingforwards, the long range lidar unit may backwards, to the right side,the left side, upwards, and/or downwards. The long range lidar unit mayhave a range greater than the other lidar units. The long range lidarunit may have a field of view that is narrower than or wider than theother lidar units. The long range lidar unit may have a field of viewthat is narrower than or wider than a collective field of view of agrouping of lidar units. The long range lidar unit may have a field ofview that is narrower than the collective field of view of all the otherlidar units.

The long range lidar unit may have a fixed position. For instance, thelong range lidar unit may remain facing in the same direction, relativeto the vehicle. The long range lidar unit may remain facing in the samedirection, relative to the other lidar units. In some alternativeembodiments, the long range lidar unit may move. The long range lidarunit may move relative to the vehicle. The long range lidar unit maymove relative to the other lidar units. The long range lidar unit maychange its orientation relative to the vehicle. The long range lidarunit may change its orientation relative to the other lidar units. Anactuator may or may not be provided that may cause the long range lidarunit to change its orientation. In some embodiments, a hinge, balljoint, pin, linkage, shaft, or other mechanical component may beprovided that may allow the long range lidar unit to change itsorientation.

The long range lidar unit may be located on top of a vehicle. The longrange lidar unit may be located at the same height as the other lidarunits of the multi-lidar module. The long range lidar unit may belocated at a different height as the other lidar units of themulti-lidar module (e.g., higher or lower). The long range lidar unitmay be located at or near the top 50%, 40%, 30%, 25%, 20%, 15%, 10%, 5%,3%, or 1% of the vehicle. The long range lidar unit may be located at ornear the bottom 50%, 40%, 30%, 25%, 20%, 15%, 10%, 5%, 3%, or 1% of thevehicle. The long range lidar unit may be located at or near the frontof the vehicle. The long range lidar unit may be located within 50%,40%, 30%, 25%, 20%, 15%, 10%, 5%, 3%, or 1% of the front of the vehicle.

The long range lidar unit may be located between two groups of lidarunits. The long range lidar unit may be located directly between and inline with two groups of lidar units. In some embodiments, data collectedby the long range lidar unit can be fused with data collected by one ormore groups of lidar units, for improved and more accurate sensing. Thefusion of data from the long range lidar unit and the one or more groupsof lidar units can also be used to create a more detailed andcomprehensive environmental map. The long range lidar unit may have adifferent support structure than the groups of lidar units.

The long range lidar unit may be used to detect remote objects. The longrange lidar unit may be used in conjunction with other long-rangingsensors (e.g., radar, cameras) for improved reliability.

FIG. 8 shows an example of multiple vision sensors being arranged on avehicle to provide a plurality of binocular cameras, in accordance withsome embodiments. The vehicle may be a land-based vehicle such as anautomobile. A vision sensing system 130 may comprise a plurality ofvision sensors 132 located on different sides of a vehicle 100. The datacollected by the vision sensors can be used to determine the presence orabsence of obstacles in an environment proximate to or surrounding thevehicle, as well as the distances of those obstacles to the vehicle. Thedata collected by the vision sensors can also be used to obtainpositional and/or motion information, that can be used to control and/orenable autonomous navigation of the vehicle.

The visual sensing range of the vision sensing system may correspond toan environmental sensing range. Each vision sensor may have an angle ofview. The maximum visual sensing range may be determined based on theangle of view (horizontally, vertically, and diagonally) and imagesensor size within each vision sensor. The angle of view defines thefield of the view (FOV) of the vision sensor. A visual sensing range canbe defined by any shape and/or size, and the sensitivity/accuracy of thevisual sensing may decrease with increasing distance away from thevehicle. In some embodiments, the visual sensing range can be defined bya regular shape (e.g., cube, cylinder, cone, etc.) or an irregular shapesurrounding the vehicle.

The vision sensing system 130 may be in operable communication with oneor more processors onboard the vehicle. Alternatively, the one or moreprocessors may be remote from the vehicle. The processors may beconfigured to execute an algorithm for selecting which vision sensorsand/or vision sensing data to utilize under various conditions, asdescribed in detail elsewhere herein. The algorithm can be implementedusing a combination of software and/or hardware. Examples of the variousconditions may include sensor malfunction, inaccuracies or deviations inthe vision sensing data, different types of environments in which thevehicle operates, etc.

The vision sensors can be any suitable device for collecting opticalsignals of the surrounding environment, that can be processed togenerate image data of the surrounding environment which may include oneor more obstacles. Any suitable number of vision sensors can be used,for example a single vision sensor, a pair of vision sensors, threevision sensors, four vision sensors, or any other number of visionsensors. A vision sensor as used herein may be referred tointerchangeably as a camera or an imaging device. In some cases, avision sensor can be an optical component of a camera or an imagingdevice. The vision sensors may be part of different imaging devices thatare capable of operating in different modes. For example, the visionsensors may be part of one or more monocular cameras and/or multi-ocularcameras. Examples of multi-ocular cameras include binocular cameras alsoknown as stereo cameras.

In some embodiments, the vision sensing system may include at least oneimaging device that is configured to operate in a monocular mode, and atleast one imaging device that is configured to operate in a multi-ocularmode. In some embodiments, a single imaging device may be configured tooperate and switch between two or more imaging modes, which may includea monocular mode and a multi-ocular mode. As an example, an imagingdevice may be configured to operate in the monocular mode in oneinstance, and operate in the multi-ocular mode in another instance. Amulti-ocular mode may include a binocular mode (or stereo mode).

Referring to FIG. 8, the vision sensing system 130 may include aplurality of vision sensors 132 supported by a vehicle 100. The visionsensors may be stereo vision sensors, and can be used collectively toform one or more binocular cameras. The vision sensors may be coupled todifferent sides of the vehicle. In some embodiments, the vision sensorscan be rigidly coupled to the vehicle such that the positions of thevision sensors are fixed relative to the vehicle. Alternatively, thevision sensors may be operably coupled to the vehicle via one or morecarriers, that permit the vision sensors to move relative to the vehiclewith respect to up to six degrees of freedom. For example, a visionsensor may be configured to tilt (e.g. pitch upwards, downwards orsideways) by a predetermined amount with aid of a carrier, therebychanging the direction of its optical axis relative to the vehicle.

The plurality of vision sensors may be laterally spaced apart ondifferent sides (e.g., front, rear, left, and right sides) of thevehicle. Each side of the vehicle may be configured to support two ormore vision sensors. The vision sensors can be separated laterally apartaround the vehicle body by up to 1 m, 500 cm, 250 cm, 100 cm, 50 cm, 25cm, 10 cm, 5 cm, 2 cm, or 1 cm. The vision sensors can be collectivelyused to provide a multi-ocular vision sensing system. For example, eachvision sensor can provide an image from a different viewpoint relativeto the vehicle, that can be used to enable stereo imaging. The visionsensors can be paired with one another in different ways andcombinations, to provide binocular (stereo) cameras having differentbaseline lengths on different sides of the vehicle body. For example,referring to FIG. 8, vision sensors 132-1 through 132-3 may be supportedon the left side of the vehicle body. The vision sensors 132-1 through132-3 may be laterally spaced apart by different distances along theleft side of the vehicle body. For example, sensors 132-1 and 132-2 maybe separated by a distance l1, and sensors 132-2 and 132-3 may beseparated by a distance l2. The distance l1 may be less than thedistance l2. Alternatively, the distance l1 may be greater than thedistance l2. In some other embodiments, the vision sensors 132-1 through132-3 may be laterally spaced apart by a same distance such that thedistances l1 and l2 are equal. Each of the distances l1 and l2 may beabout 0.1 m, 0.2 m, 0.3 m, 0.4 m, 0.5 m, 0.6 m, 0.7 m, 0.8 m, 0.9 m, 1m, or greater.

The vision sensors can be combined in different ways to form binocularcameras having different baseline lengths. The binocular cameras maycomprise different subsets of vision sensors. In some embodiments, afirst binocular camera may comprise a first subset of vision sensorscomprising at least two vision sensors having a first baselinetherebetween, and a second binocular camera may comprise a second subsetof vision sensors comprising at least two vision sensors having a secondbaseline therebetween. For example, referring to FIG. 8, a firstbinocular camera 134-1 may comprise of vision sensors 132-1 and 132-2having a first baseline b1 defined by the distance l1 therebetween.Likewise, a second binocular camera 134-2 may comprise of vision sensors132-2 and 132-3 having a second baseline b2 defined by the distance l2therebetween. The first baseline b1 may be shorter than the secondbaseline b2. Alternatively, the first baseline b1 may be equal to orgreater than the second baseline b2. In some embodiments, a thirdbinocular camera 134-3 may comprise of vision sensors 132-1 and 132-3having a third baseline b3 defined by the sum of distances 11 and 12. Asshown in FIG. 8, one or more of the binocular cameras 134 may utilizeone or more common vision sensors. For example, the vision sensor 132-2may be utilized by the first and second binocular cameras, and mayfunction as the “left-eye” of the first binocular camera 134-1 andfunction as the “right-eye” of the second binocular camera 134-2 whenthese cameras are imaging from the left side of the vehicle body.Alternatively, the vision sensor 132-1 may be utilized by the first andthird binocular cameras, and may function as the “right-eye” of each ofthe first and third binocular cameras 134-1 and 134-3 when these camerasare imaging from the left side of the vehicle body. In other embodimentsdescribed elsewhere herein, the plurality of binocular cameras need notshare any common vision sensors, and each binocular camera may comprisea unique subset (or pair) of vision sensors.

Accordingly, a plurality of different binocular cameras having differentbaselines can be provided on multiple sides of the vehicle body,depending on the number of vision sensors coupled to the vehicle bodyand the lateral distance(s) between the vision sensors. The binocularcameras can be disposed on the same side of the vehicle, on laterallyadjacent sides of the vehicle, or on opposite sides of the vehicle. Forexample, one or more binocular cameras can be disposed on the front,rear, or lateral sides of the vehicle, or suitable combinations thereof.

In some embodiments, binocular cameras located on different sides (e.g.laterally adjacent sides) of the vehicle body may share one or morecommon vision sensors. For example, referring to FIG. 8, a plurality ofbinocular cameras may be provided on the front side of the vehicle body.A fourth binocular camera 134-4 may comprise of vision sensors 132-1 and132-4 having a fourth baseline b4 defined by the distance l4therebetween. The vision sensor 132-1 may be shared by the first andfourth binocular cameras 134-1 and 134-4, and may be capable of rotatingabout a vertical axis to change the direction of its optical axis.

For example, when the vision sensor 132-1 is oriented such that itsoptical axis OA is substantially perpendicular to the left side of thevehicle body, the vision sensor 132-1 may function as the “right-eye” ofthe first binocular camera 134-1. The vision sensor 132-1 may be capableof rotating about a vertical axis such that its optical axis can bealigned at different angles relative to the vertical axis. When thevision sensor 132-1 is oriented such that its optical axis OA′ issubstantially perpendicular to the front side of the vehicle body, thevision sensor 132-1 may then function as the “left-eye” of the fourthbinocular camera 134-1. As such, a single vision sensor can be utilizedin different binocular cameras located on different sides of the vehiclebody, depending on the direction in which its optical axis is oriented.

The vision sensors 132 can be configured to acquire a plurality ofimages from different positions or viewpoints relative to the vehicle.An object that is proximal to the vision sensors generally has a largerparallax compared to an object that is further away from the visionsensors. A binocular error δ for a given subset of vision sensors in abinocular camera may be given by

$\delta = {\frac{z^{2}}{fb}ɛ}$

where z corresponds to a depth of the imaged object from an image plane,f is the focal length of the vision sensors, b is the length of thebaseline, and ε is the parallax. The binocular error δ may be indicativeof an error in the depth of the imaged object as extracted from one ormore stereoscopic images captured by the binocular camera. A highbinocular error indicates a large error in the “perceived” distance ofthe object(s) from the vehicle, whereas a low binocular error indicatesa low error in the “perceived” distance of the object(s) from thevehicle. For objects that are located further away (i.e. z is greater),parallax ε is low and binocular error δ can be reduced by increasing thelength of the baseline b. The length of the baseline b can be increased,for example by increasing the lateral distance between the visionsensors. The increase in baseline b also increases disparity whichallows more accurate depth and distance information to be obtained.

For objects that are located closer to the vision sensors, parallax ε ishigher but the binocular error δ can be maintained by using a shorterbaseline b since z is lower. Thus, a binocular camera comprising asubset of vision sensors that are spaced further apart (i.e. longerbaseline b) can be useful for imaging distant objects, since thebinocular error is lower and it is more likely for distant objects tofall within its field of view. However, due to the increased baseline,proximal objects may not fall within the field of view of theabovementioned binocular camera.

To address the above, a binocular camera comprising a pair of visionsensors that are spaced closer together (i.e. shorter baseline b) can beuseful for imaging proximal objects, since the binocular error is lowerand it is more likely for proximal objects to fall within its field ofview. Referring to FIG. 8, the first baseline b1 of the first binocularcamera 134-1 may be less than the second baseline b2 of the secondbinocular camera 134-2. Accordingly, the first binocular camera 134-1comprising of the vision sensors 132-1 and 132-2 can be configured toobtain depth information of objects that are located closer to the leftside of the vehicle body, whereas the second binocular camera 134-2comprising of the vision sensors 132-2 and 132-3 can be configured toobtain depth information of objects that are located further away fromthe left side of the vehicle body. Additionally or optionally, anotherbinocular camera comprising of the vision sensors 132-2 and 132-4 can beconfigured to obtain depth information of objects that are locatedfurther away from the left side of the vehicle body.

The first and second binocular cameras can be configured to capturestereoscopic images. One or more processors onboard the vehicle orremote from the vehicle may be configured to process the stereoscopicimages, and calculate a binocular error δ1 for the first binocularcamera 134-1 and a binocular error δ2 for the second binocular camera134-2. The processors may be further configured to compare the binocularerrors δ1 and δ2 to determine which binocular camera (and correspondingset of stereoscopic images) to utilize for obtaining depth informationof objects located at different distances from the cameras. For example,when δ1 is greater than δ2, the processors may select and utilize thesecond binocular camera 134-2 over the first binocular camera 134-1 forvision sensing, since the imaged object(s) may be located further awayfrom the left side of the vehicle body. Conversely, when δ2 is greaterthan δ1, the processors may select and utilize the first binocularcamera 134-1 over the second binocular camera 134-2 for vision sensing,since the imaged object(s) may be located closer to the left side of thevehicle body. In some embodiments, when δ1 is equal to δ2, theprocessors may select either the first binocular camera or the secondbinocular camera for vision sensing since the binocular error is thesame for both cameras.

In some embodiments, the processors can be configured to determine afirst disparity d1 between matched points in stereoscopic imagescaptured by the first binocular camera 134-1, and a second disparity d2between matched points in stereoscopic images captured by the secondbinocular camera 134-2. The processors can compare the first and seconddisparities d1 and d2 to determine which binocular camera (andcorresponding set of stereoscopic images) to utilize for obtaining depthinformation of objects located at different distances from the cameras.For example, when d1 is greater than d2, the processors may select andutilize the first binocular camera 134-1 over the second binocularcamera 134-2 for vision sensing, since more accurate depth informationcan be extracted from the stereoscopic images captured by the firstbinocular camera. Conversely, when d2 is greater than d1, the processorsmay select and utilize the second binocular camera 134-2 over the firstbinocular camera 134-1 for vision sensing, since more accurate depthinformation can be extracted from the stereoscopic images captured bythe second binocular camera. In some embodiments, when d1 is equal tod2, the processors may select either the first binocular camera or thesecond binocular camera for vision sensing since the disparity is thesame for both.

In some embodiments, the processors may be configured to compare thefirst and second disparities d1 and d2 to a predefined thresholddisparity dp, in order to determine which binocular camera (andcorresponding set of stereoscopic images) to utilize. The predefinedthreshold disparity can be obtained from experimental data. Thepredefined threshold disparity may be a single value or may comprise arange of values. In some embodiments, the predefined threshold disparitycan be configured to vary or be adjusted depending on one or more of thefollowing: (1) the environment in which the vehicle is beingautonomously operated, (2) weather conditions within the environment,(3) an altitude of the vehicle, (4) object density and distributionwithin the environment, or (5) visual or physical properties of objectslocated within the environment.

In some cases, when (1) d1 is greater than the dp and (2) d2 is lessthan dp, the processors may select and utilize the first binocularcamera over the second binocular camera for vision sensing since moreaccurate depth and distance information can be obtained using the firstbinocular camera. Conversely, when (1) d2 is greater than the dp and (2)d1 is less than dp, the processors may select and utilize the secondbinocular camera over the first binocular camera for vision sensingsince more accurate depth and distance information can be obtained usingthe second binocular camera. When both d1 and d2 are greater than dp,the processors may compare d1 and d2 to determine which is higher, andselect the binocular camera having the higher disparity for visionsensing and depth extraction. In some cases, both d1 and d2 may be lessthan dp, for example in an environment with poor lighting or adverseweather conditions. In those cases, the processors may not select thevision sensors for environmental sensing and depth extraction, and mayinstead select or utilize another set of non-vision sensors (e.g.,lidar, radar or ultrasonic) for sensing the surrounding environment.

The vision sensors can be used to simultaneously capture images at aspecified frequency to produce a time series of image data. The timeseries of image data obtained from the vision sensors can be processedto determine the position, orientation, and/or velocity of the vehicleusing any suitable method, such as a machine vision algorithm. Forexample, a machine vision algorithm can be used to identify one or morefeature points within each image (e.g., an edge of an object, a cornerof an object, or a boundary between objects of two different colors).Any suitable method or combination of methods can be used to identifyand provide a digital representation of the feature points, such as thefeatures from accelerated segment test (FAST) algorithm or the binaryrobust independent elementary features (BRIEF) algorithm. The image datacan then be matched to each other to identify a set of common featurepoints appearing in images obtained by both vision sensors. The motionof the vehicle can be determined based on the common feature points andthe spatial disposition of the vision sensors relative to the vehicleand to each other.

As previously described, an optical axis of a vision sensor may beperpendicular to the side of the vehicle body on which the vision sensoris located. In some embodiments, a vision sensor may be capable ofrotating about a vertical axis such that its optical axis extend fromthe side of the vehicle body in a non-orthogonal manner (e.g., at anacute angle or oblique angle). The vision sensors may be configured torotate (and/or translate) to focus at different points in space.

FIG. 9 shows an example of multiple binocular cameras being arranged ona vehicle for sensing various directions and ranges, in accordance withan embodiment of the disclosure. In FIG. 9, the vision sensors 132-1 and132-2 of the first binocular camera 134-1 may be oriented (e.g. rotated)such that their respective optical axes intersect at point A in space.In another example, the vision sensors 132-2 and 132-3 of the secondbinocular camera 134-2 may be oriented (e.g. rotated) such that theirrespective optical axes intersect at point B in space. In a furtherexample, the vision sensors 132-1 and 132-3 of the third binocularcamera 134-3 may be oriented (e.g. rotated) such that their respectiveoptical axes intersect at point C in space. Likewise, in anotherexample, the vision sensors 132-1 and 132-4 of the fourth binocularcamera 134-4 may be oriented (e.g. rotated) such that their respectiveoptical axes intersect at point D in space. The above binocular camerashave different disparities which are relative to the vergence angles.For any angle, there is a surface in space corresponding to zerodisparity. For example, point A lies on a surface S(A) of zero disparityfor the first binocular camera, point B lies on a surface S(B) of zerodisparity for the second binocular camera, and point C lies on a surfaceS(C) of zero disparity for the third binocular camera. These zerodisparity surfaces S(A), S(B), and S(C) are located at differentdistances from the vehicle body. For example, the surface (A) may belocated closest to the vehicle, the surface S(C) may be located furthestaway from the vehicle, and the surface S(B) may be located betweensurfaces S(A) and S(B).

For a given surface, objects that are farther away from the surface havedisparity greater than zero, and objects that are before the surfacehave disparity less than zero. Within a region, the disparities can begrouped into three pools:

Disparities + d > 0 − d < 0 0 d = 0

The difference in disparities in stereoscopic images obtained bydifferent binocular cameras can be used to resolve any ambiguous matchesfor objects located at different distances from the vehicle. Thedifference in disparities can also be used to determine which zone(relative to the vehicle) that an obstacle lies in. For example, in FIG.9, a plurality of obstacles 105, 106, 107 and 108 may be located atdifferent distances from the left side of the vehicle body. One or moreobstacles may lie within the surface of zero disparity for a particularbinocular camera, but outside of the surface of zero disparity ofanother binocular camera. The differences in disparities can besummarized in the table below:

Binocular Surface S of Obstacle Obstacle Obstacle Obstacle camera zerodisparity 105 106 107 108 134-1 S(A) d < 0 d > 0 d > 0 d > 0 134-2 S(B)d < 0 d < 0 d > 0 d > 0 134-3 S(C) d < 0 d < 0 d < 0 d > 0

As described above with reference to FIG. 9, the vision sensors can beoriented in various configurations to focus at different points inspace. In some embodiments, the vision sensors may be capable ofshifting their optical axes in real-time to create different surfaces ofzero disparity, to extract depth information of objects located atvarious distances from the vehicle body, and also to resolve anyambiguities in the extracted depth information. For example, the depthinformation obtained by at least one binocular camera can be comparedwith the depth information obtained by one or more other binocularcameras to correct for binocular errors.

FIG. 10 shows an example of a vehicle with a plurality of binocularcameras comprising various combinations of vision sensors, in accordancewith an embodiment of the disclosure. The vision sensing system maycomprise a plurality of vision sensors coupled to different sides of thevehicle body. For example, the plurality of vision sensors can becoupled the vehicle body such that the optical axes and field of view ofthe vision sensors extend from different sides (e.g., front, rear, leftand right sides) of the vehicle body. In the example of FIG. 10, thevision sensors may be mounted on or integrated into a top portion (e.g.hood) of the vehicle.

The vision sensing system of FIG. 10 operates in a similar manner to thesystem of FIG. 8 except for the following differences. In FIG. 10, eachbinocular camera 135 is comprised of a unique subset of vision sensors133, and need not share any vision sensor with another binocular camera.Accordingly, all of the binocular cameras 135 in FIG. 10 are capable ofoperating independently and simultaneously to image the environmentproximate to or surrounding the vehicle body. In some cases, the visionsensors 133 may be rigidly coupled to the vehicle body such that theymaintain a same field of view with respect to the side(s) of the vehiclebody. In some alternative embodiments (not shown), one or more of thevision sensors 133 may be capable of changing its orientation to changethe direction of its optical axis, as described elsewhere herein withrespect to FIG. 9.

In FIG. 10, a first binocular camera 135-1 and a second binocular camera135-2 are supported facing out from the right side of the vehicle body.The first binocular camera 135-1 may be comprised of vision sensors133-1 and 133-2 having a first baseline b1′ defined by the distance l1′therebetween. Likewise, the second binocular camera 135-12 may becomprised of vision sensors 133-3 and 133-4 having a second baseline b1′defined by the distance l1′ therebetween. Unlike FIG. 8, the first andsecond binocular cameras 135-1 and 135-2 have unique subsets of visionsensors, and need not utilize any common vision sensor. In someembodiments, the pair of vision sensors of the second binocular cameramay be positioned between the pair of vision sensors of the firstbinocular camera on one side of the vehicle. The second binocular cameramay be configured to collect image data closer to the vehicle than thefirst binocular camera, and within a blind spot of the first binocularcamera.

As shown in FIG. 10, the first baseline b1′ may be longer than thesecond baseline b2′. In some alternative embodiments, the first baselineb1′ may be less than the second baseline b2′. The vision sensors 133have overlapping fields of view to ensure that sufficient image datapoints of the surrounding environment can be collected. The firstbinocular camera 135-1 has a field of view that is greater than thesecond binocular camera 135-2. Accordingly, the first binocular camera135-1 comprising the pair of vision sensors 133-1 and 133-2 that arespaced further apart (i.e. longer baseline b) can be useful for imagingdistant objects, since the binocular error is lower and it is morelikely to capture distant objects within its field of view. In contrast,the second binocular camera 135-2 comprising the pair of stereo visionsensors 133-3 and 133-4 that are spaced closer together (i.e. shorterbaseline b) can be useful for imaging proximal objects, since thebinocular error is lower and it is more likely to capture proximalobjects within its field of view. As previously described, differentbinocular cameras of different baselines can be configured to imageobjects located at different distances from the corresponding side(s) ofthe vehicle body, in order to minimize binocular errors and improve theaccuracy of the extracted depth information. An environmental map can beconstructed with a certain level of accuracy based on the collectedimage data points, as described elsewhere herein.

The vision sensors shown in FIGS. 8-10 may be configured to capturebinocular (stereoscopic) or multi-ocular images of the environmentproximate to or surrounding the vehicle. Additionally or optionally, oneor more monocular cameras may be provided on the vehicle, and configuredto capture monocular color images. One or more of the cameras maycapture images at a same time instance or at different time instances. A3-D depth map of the environment can be obtained from the binocular ormulti-ocular images. The plurality of vision sensors may provide fieldsof view of n degrees. In some embodiments, n may be about 90°, 100°,110°, 120°, 130°, 140°, 150°, 160°, 170°, 180°, 190°, 200°, 210°, 220°,230°, 240°, 250°, 260°, 270°, 280°, 290°, 300°, 310°, 320°, 330°, 340°,350°, or 360°. Any value for n may be contemplated. For example, n maybe greater than 0°, or less than or equal to 360°. When n is 360°,complete-surround visual sensing can be obtained. In some cases, thevisual sensing range may be defined by any shape having a predetermineddistance from the center of the vehicle. The predetermined distance mayrange from several meters to hundreds of meters. For example, thepredetermined radius may be about 1 m, 5 m, 10 m, 20 m, 30 m, 40 m, 50m, 60 m, 70 m, 80 m, 90 m, 100 m, 200 m, 300 m, 400 m, 500 m, or anyvalues therebetween. In some cases, the predetermined distance may beless than 1 m or greater than 500 m. Any value for the predeterminedradius may be contemplated. In some embodiments, the visual sensingrange may depend on an environmental complexity of the environment inwhich the vehicle operates. The visual sensing range can dynamicallyadjust as the vehicle moves through different environments. For example,when the vehicle is moving in an environment comprising a large numberof objects or obstacles, the visual sensing range can be extended,and/or a sensitivity level (e.g., resolution) of the visual sensing maybe increased. Conversely, when the vehicle is moving in an environmentcomprising a low number of objects or obstacles, the visual sensingrange may be reduced, and/or a sensitivity level (e.g., resolution) ofthe visual sensing may be decreased.

The cameras may be capable of taking multiple images substantiallysimultaneously, sequentially, or at different points in time. Themultiple images may aid in the creation of a 3D scene, a 3D virtualenvironment, a 3D map, or a 3D model. For instance, a right-eye imageand a left-eye image may be taken and used for stereo-mapping. A depthmap may be calculated from a calibrated binocular image, as described indetail below. Any number of images (e.g., 2 or more, 3 or more, 4 ormore, 5 or more, 6 or more, 7 or more, 8 or more, 9 or more) may betaken simultaneously to aid in the creation of a 3D scene/virtualenvironment/model, and/or for depth mapping. The images may be directedin substantially the same direction or may be directed in slightlydifferent directions. In some instances, data from other sensors (e.g.,ultrasonic data, LIDAR data, data from any other sensors as describedelsewhere herein, or data from external devices) may aid in the creationof a 2D or 3D image or map.

In some embodiments, stereoscopic image data obtained from one or morecameras can be analyzed to determine the environmental information. Thestereoscopic image data can include binocular images or video. Theenvironmental information may comprise an environmental map. Theenvironmental map may comprise a topological map or a metric map. Themetric map may comprise at least one of the following: a point cloud, a3D grid map, a 2D grid map, a 2.5D grid map, or an occupancy grid map.The occupancy grid may be used to define a 3D map of the spatialenvironment proximate to or surrounding the movable object.

In some embodiments, analysis of stereoscopic video data may comprise atleast one of the following: (1) imaging device calibration, (2) stereomatching of image frames, and (3) depth map calculation. The imagingdevice calibration may comprise calibrating intrinsic parameters andextrinsic parameters of an imaging device such as a binocular camera.The binocular camera may be configured to capture one or more binocularimages. The stereoscopic video data may be obtained from a plurality ofbinocular images. The stereo matching may comprise (1) extractingsubstantially in or near real-time feature points of each monocularimage in each binocular image, (2) calculating the motioncharacteristics of the feature points, (3) matching correspondingfeature points extracted from the image frames based on the motioncharacteristics of the feature points, and (4) eliminating mismatchfeature points. The depth map calculation may comprise (1) calculating apixel-based disparity map based on the matched feature points and (2)calculating a depth map based on the extrinsic parameters of thebinocular camera. The depth map calculation may comprise filtering andapplying a threshold to the depth map to determine or more obstacles.For example, the threshold may be applied to classify objects in theenvironment having a predetermined size and/or number of pixels in thedepth map.

FIG. 11 illustrates a binocular camera 134 for stereo vision, inaccordance with some embodiments. The binocular camera can include aleft vision sensor and a right vision sensor (not shown) centered atpositions 1102 and 1104, respectively. The parameters focal length f,photosensor size l, and the baseline distance b between the visionsensors are known for the binocular camera. The 3D coordinate{circumflex over (p)}_(t) ^(i) corresponds to a pair of matched featurepoints m_(t) ^(i)=(u_(t) ^(i), v_(t) ^(i)) and m_(t) ^(i)′=(u_(t) ^(i)′,v_(t) ^(i)) in the images captured by the left and right vision sensors,respectively. The pixel distances u_(t) ^(i) and u_(t) ^(i)′ can bemultiplied by the size of a single pixel to become spatial distancesū_(t) ^(i) and ū_(t) ^(i)′. Thus, using the formula

${{\frac{D}{f}{{{\overset{\_}{u}}_{t}^{i} - {l/2}}}} + {\frac{D}{f}{{{\overset{\_}{u}}_{t}^{i\prime} - {l/2}}}}} = b$

the distance between the 3D coordinate {circumflex over (p)}_(t) ^(i)and the vision sensors, denoted D, can be determined. Based on theinternal parameter matrix of the camera K and calculated value of D, theestimated 3D coordinate {circumflex over (p)}_(t) ^(i) can thus bederived for the point (u_(t) ^(i), v_(t) ^(i)).

Following frame-to-frame matching and stereo matching of feature points,a feature point-3D coordinate pair c_(i)={m_(t) ^(i), {circumflex over(p)}_(t) ^(i)} can be obtained for each feature point. The velocity ofthe camera can thus be determined by analyzing the motion of the featurepoints within the images using any suitable algorithm. For example,given a set of n coordinate pairs c₁, c₂, . . . , c_(n) obtained at atime t, the matrix {tilde over (R)} can be expressed as three rowvectors {tilde over (R)}=[{tilde over (r)}₁ {tilde over (r)}₂ {tildeover (r)}₃]^(T), and the internal parameter matrix of the camera can beexpressed as

$K = \begin{bmatrix}f_{u} & 0 & u_{c} \\0 & f_{v} & v_{c} \\0 & 0 & 1\end{bmatrix}$

Consequently, an estimated positional movement or change between eachfeature point in time, {tilde over (T)}_(v), can be obtained by solving

$\begin{bmatrix}{{( {u_{t}^{1} - u_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{1}} - {f_{u}{\overset{\sim}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{1}}} \\{{( {v_{t}^{1} - v_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{1}} - {f_{v}{\overset{˜}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{1}}} \\{{( {u_{t}^{2} - u_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{2}} - {f_{u}{\overset{˜}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{2}}} \\{{( {v_{t}^{2} - v_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{2}} - {f_{v}{\overset{˜}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{2}}} \\\ldots \\{{( {u_{t}^{n} - u_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{n}} - {f_{u}{\overset{˜}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{n}}} \\{{( {v_{t}^{n} - v_{c}} ){\overset{˜}{r}}_{3}^{T}{\overset{\hat{}}{p}}_{t}^{n}} - {f_{v}{\overset{˜}{r}}_{1}^{T}{\overset{\hat{}}{p}}_{t}^{n}}}\end{bmatrix} = {\begin{bmatrix}f_{u} & 0 & {u_{c} - u_{t}^{1}} \\0 & f_{v} & {v_{c} - v_{t}^{1}} \\f_{u} & 0 & {u_{c} - u_{t}^{2}} \\0 & f_{v} & {v_{c} - v_{t}^{2}} \\\ldots & \ldots & \ldots \\f_{u} & 0 & {u_{c} - u_{t}^{n}} \\0 & f_{v} & {v_{c} - v_{t}^{n}}\end{bmatrix} \cdot {\overset{\_}{T}}_{v}}$

As the estimated positional movement is primarily obtained based onframe matching of image data from the left and right vision sensors, theaccuracy and precision of this estimate can be influenced by the size ofn. For example, the accuracy and precision of the estimate may increaseas n increases.

The vision sensors described in FIGS. 8-10 may be characterized by oneor more sensor parameters. The sensor parameters may be intrinsic orextrinsic parameters. An intrinsic parameter may relate to the internalconfiguration of a vision sensor. Example of intrinsic parameters mayinclude focal length, scale factor, radial distortion coefficients, andtangential distortion coefficients. Intrinsic parameters may be anyparameters that are dependent on hardware configurations, in some casesthe intrinsic parameters may be set by a factory setting for the visionsensor. Extrinsic parameters may relate to the spatial relationshipbetween any two or more vision sensors. Each vision sensor may have arelative coordinate system independent of other sensors on board themovable object. Extrinsic properties may be important for sensor fusion,combining data from sensors in different locations on the movableobject. Sensor fusion may involve a process of transforming the relativecoordinates of a given sensor to match the reference frame of anothersensor.

FIG. 12 illustrates the transforming of relative coordinates of one ormore cameras to match the reference frame of the vehicle. Atransformation may be conducted such that the coordinate system of eachcamera is rotated to match the coordinate system of the vehicle body.The transformation may be accounted for mathematically by atransformation matrix.

Intrinsic properties may be specific to a sensor and may varyinfrequently. Recalibration of intrinsic properties may occurperiodically while the vehicle is not operating. It may not be criticalto recalibrate intrinsic properties while the vehicle is operatingbecause these properties remain relatively consistent while the vehicleis operating in comparison to extrinsic properties. Intrinsic propertiesmay be calibrated by interpreting an image of a known calibrationstandard or target. Vanishing lines or points on a calibration standardor target may be used to calibrate the intrinsic properties such asfocal length and distortion.

Extrinsic properties may change with a higher frequency compared tointrinsic properties. Shifting during stop and motion of the vehicle,vibration, and thermal drift may cause changes in the extrinsicproperties of the sensors. For example a camera location may shift dueto vibration of the vehicle during driving. Extrinsic properties maydeviate from their initial configuration while the vehicle is inoperation; therefore it may be preferably to perform a recalibration ofthe extrinsic properties while the vehicle is operating. Recalibrationof the extrinsic properties while the vehicle is operating may requirecomputing resources. The computing resources may be onboard or off-boardthe vehicle.

The recalibration of extrinsic properties may occur with a set timefrequency for example, the extrinsic properties may be recalibratedevery 1 min, 5 min, 10 min, 20 min, 30 min, 40 min, 50 min, 1 hour, 2hours, 3 hours, 4 hours, 5 hours, 10 hours, 12 hours, or once every day.Alternatively the recalibration of extrinsic properties may occur with aset distance frequency, for example a recalibration may occur every timethe vehicle travels an additional 0.5 mile, 1 mile, 2 miles 3 miles, 4miles, 5 miles, 10 miles, 15 miles, 20 miles, 25 miles, 30 miles, 35miles, 40 miles 45 miles, 50 miles, or 100 from an initial startinglocation. The frequency of the onboard calibration of extrinsicproperties may be decided based on the available computing resources,fuel or power requirements, terrain and/or weather conditions. Someconditions may decrease or increase the expected drift in thecalibration of the extrinsic sensors, for example, if the vehicle isdriving at a low speed on a smooth paved road, there may be lessvibration of the body of the vehicle and therefore less drift in thecalibration of the extrinsic sensors.

The extrinsic parameters may have an initial calibration. The initialcalibration of the extrinsic parameters may describe relativedifferences between the sensors, e.g., relative locations, rotations,and/or displacements of two or more sensors. The parameters may includechanges to sensors over time, such as displacements of sensors between acertain time and a subsequent time. The displacements may includetranslational displacement and/or rotational displacement. Thetranslational displacement may occur along one or more of the 3 axes.Similarly, the rotational displacement may occur in one or more of the 3axes. In general, the calibration is achieved by a filtering process;non-limiting examples include various types of Kalman filters.

The calibration of the extrinsic parameters may be adjusted while thevehicle is in operation (e.g. during driving). A method of calibratingthe extrinsic parameters may comprise detecting, with aid of aprocessor, a change in a spatial configuration of two or more sensorsrelative to one another from the initial spatial configuration to asubsequent spatial configuration. In further embodiments, the methoddetermines the subsequent spatial configuration using filters, such asKalman filters. Finally, the method may include adjusting data from atleast one of the sensors while the vehicle is in motion based on thesubsequent spatial configuration.

The vehicle may have one or more on board processors. The processors maybe individually or collectively, configured to (i) detect a change in aspatial configuration of one or more sensors relative to one anotherfrom the initial spatial configuration to a subsequent spatialconfiguration, based on the sensor data; (ii) determine the subsequentspatial configuration using a plurality of Kalman filters; and (iii)adjust data, while the vehicle is in motion, from at least one of thesensors based on the subsequent spatial configuration. Alternatively theprocessor or processors may be off board the vehicle. The vehicle maytransmit information about the spatial configuration of a given sensorto an off board processor which may be configured to perform theaforementioned steps (i)-(iii) and transmit the information back to thevehicle.

In some embodiments, multiple cameras can be calibrated relative to oneanother. Calibrating multiple vision sensors may comprise integratingsensor data. A first camera can capture a first image, and a secondcamera can capture a second image with different displacement anddifferent orientation. Therefore, the two cameras need to be calibrated,and the calibration can utilize both the first image taken by the firstcamera and the second image taken by the second camera. The followingwill disclose the mathematical formulation of calibrations.

In an embodiment, two or more cameras are assembled into a stereo camerasystem. The calibration of the two or more cameras is as follows. First,each camera takes an image. Then, an identification system selects Nfeatures. In terms of mathematical formulation, let α and β denote twocameras. The features identified in their images are denoted by vectorsx_(i) ^(α)=(x_(i) ^(α), y_(i) ^(α)) and x_(i) ^(β)=(x_(i) ^(β), y_(i)^(β)), where i=1, . . . , N. The features x_(i) ^(α) and x_(i) ^(β) aredetermined by the coordinate systems of cameras α and β, respectively.To find a faithful mapping, the features need to be analyzed in a samereference coordinate system X_(i)=(X_(i), Y_(i), Z_(i)). Therelationship between features x_(i) ^(α) and x_(i) ^(β) and thereference coordinate system X_(i) can be described by projection: {tildeover (x)}_(i) ^(α)≈P^(α){tilde over (X)}_(i) and {tilde over (x)}_(i)^(β)≈P^(β){tilde over (X)}_(i), where {tilde over (x)}_(i) ^(α) and{tilde over (x)}_(i) ^(β) are features described in the normalizedcoordinates, namely {tilde over (x)}_(i) ^(α)=(x_(i) ^(α),y_(i) ^(α),1)and {tilde over (x)}_(i) ^(β)=(x_(i) ^(β),y_(i) ^(β),1). P^(α) and P^(β)are projections of cameras α and β, respectively, and they can bedetermined by the intrinsic parameters K and extrinsic parameters (e.g.,rotation R and translation T): P^(α)=K^(α)[R^(α)T^(α)] andP^(β)=K^(β)[R^(β)T^(β)]. Once the projections P^(α) and P^(β) arecomputed and the intrinsic parameters K^(α) and K^(β) are known, theextrinsic parameters R and T can be computed:

R=R ^(β)(R ^(α))⁻¹,

T=T ^(β) −R ^(β)(R ^(α))⁻¹ T ^(α)

When parameters R and T are derived, the calibration is complete.

Typically, the intrinsic parameters K^(α) and K^(β) do not change; evenif they change, the amount of change is small. Therefore, the intrinsicparameters can be calibrated off-line. Namely, in some applications, theintrinsic parameters can be determined before the UAV takes off. Ininstances, the intrinsic parameters K^(α) and K^(β) remain static duringvehicle motion, so the calibration is to compute optimal solutions forthe Pa and PP. An example uses minizing projection errors to findsolutions:

$\min\limits_{P^{\alpha},P^{\beta}}{\sum\limits_{i = 1}^{N}\lbrack {( {{\overset{\sim}{x}}_{i}^{\alpha} - {P^{\alpha}{\overset{\sim}{X}}_{i}}} )^{2} + ( {{\overset{\sim}{x}}_{i}^{\beta} - {P^{\beta}{\overset{\sim}{X}}_{i}}} )^{2}} \rbrack}$

This problem is a non-linear optimization problem. Various solutionmethods can be included in the embodiments. In some applications,solutions are achieved by bundle adjustment method. In the bundleadjustment method, projections P^(α) and P^(β) are given initial values.Using epipolar constraint to derive essential matrix E, followed by adecomposition (e.g., singular value decomposition) that obtainsE=└T┘_(x)R where └T┘_(x) is the skew symmetric matrix of T.

This solution finds a corresponding mapping between these features andthe features in another image taken by another camera. In someembodiments, the spatial configuration of two cameras α and β forming astereo camera system arranges one camera on the left hand side and theother on the right hand side.

In some embodiments, the vision sensing system may include one or moremonocular cameras. Each monocular camera may comprise a vision sensor.The monocular cameras can be operably coupled to different sides (e.g.front, left, or lateral sides) of a vehicle. In some embodiments, amonocular camera can be mounted to the vehicle via a carrier, thatpermits the monocular camera to move relative to the vehicle withrespect to up to six degrees of freedom. Alternatively, the monocularcamera can be directly mounted onto the vehicle, or coupled to a supportstructure mounted onto the vehicle. In some embodiments, the monocularcamera can be an element of a payload of the vehicle. The monocularcameras can be configured to capture image data of the environmentproximate to or surrounding the vehicle.

FIG. 13 illustrates a plurality of monocular cameras supported ondifferent sides of a vehicle, in accordance with some embodiments. Afirst monocular camera 138-1 may be mounted on the front side of thevehicle body, and a second monocular camera 138-2 may be mounted on therear side of the vehicle body. The first and second monocular camerasmay have the same imaging resolution or different imaging resolutions. Avisual detection range of the camera may be based in part on the fieldof view and the imaging resolution of the camera. For example, the imagedata captured by the first monocular camera can be used to detectobjects located up to a maximum distance d6 from the front side of thevehicle body. Similarly, the image data captured by the second monocularcamera can be used to detect objects located up to a maximum distance d7from the front side of the vehicle body. In some embodiments, thedistance d6 may be greater than d7 when the first monocular camera has ahigher imaging resolution than the second monocular camera. For example,the distance d6 may be greater than d7 by at least 10%, 20%, 30%, 40%,50%, 60%, 70%, 80%, 90%, 100%, or more than 100%. In some embodiments,the first monocular camera may be configured to capture 4K resolutionimages/video, and the second monocular camera may be configured tocapture 1080p resolution images/video. The image data (4K resolution)captured by the first monocular camera may be used to detect objectslocated up to a maximum distance of 200 m from the front side of thevehicle body. The image data (1080p resolution) captured by the secondmonocular camera may be used to detect objects located up to a maximumdistance of 100 m from the rear side of the vehicle body.

A target object (e.g., an obstacle) may be identified from 2D imagescaptured by the monocular cameras. In some cases, the target object maybe selected based on moving target detection. In those instances, thevehicle and the surrounding environment are assumed to bestatic/stationary, and the target object to be tracked may be the onlymoving object in the image. The target object can be identified andselected through background subtraction.

Identification of a target object from a 2D image may be based onfeature point recognition. A feature point can be a portion of an image(e.g., an edge, corner, interest point, blob, ridge, etc.) that isuniquely distinguishable from the remaining portions of the image and/orother feature points in the image. Optionally, a feature point may berelatively invariant to transformations of the imaged object (e.g.,translation, rotation, scaling) and/or changes in the characteristics ofthe image (e.g., brightness, exposure). A feature point may be detectedin portions of an image that is rich in terms of informational content(e.g., significant 2D texture). A feature point may be detected inportions of an image that are stable under perturbations (e.g., whenvarying illumination and brightness of an image).

Feature points can be detected using various algorithms (e.g., texturedetection algorithm) which may extract one or more feature points fromimage data. The algorithms may additionally make various calculationsregarding the feature points. For example, the algorithms may calculatea total number of feature points, or “feature point number.” Thealgorithms may also calculate a distribution of feature points. Forexample, the feature points may be widely distributed within an image(e.g., image data) or a subsection of the image. For example, thefeature points may be narrowly distributed within an image (e.g., imagedata) or a subsection of the image. The algorithms may also calculate aquality of the feature points. In some instances, the quality of featurepoints may be determined or evaluated based on a value calculated byalgorithms mentioned herein (e.g., FAST, Corner detector, Harris, etc).

The algorithm may be an edge detection algorithm, a corner detectionalgorithm, a blob detection algorithm, or a ridge detection algorithm.In some embodiments, the corner detection algorithm may be a “Featuresfrom accelerated segment test” (FAST). In some embodiments, the featuredetector may extract feature points and make calculations regardingfeature points using FAST. In some embodiments, the feature detector canbe a Canny edge detector, Sobel operator, Harris &Stephens/Plessy/Shi-Tomasi corner detection algorithm, the SUSAN cornerdetector, Level curve curvature approach, Laplacian of Gaussian,Difference of Gaussians, Determinant of Hessian, MSER, PCBR, orGrey-level blobs, ORB, FREAK, or suitable combinations thereof.

In some embodiments, a feature point may comprise one or morenon-salient features. As used herein, non-salient features may refer tonon-salient regions or non-distinct (e.g., non-recognizable) objectswithin an image. Non-salient features may refer to elements within animage that are unlikely to stand out or catch attention of a humanobserver. Examples of non-salient features may include individual pixelsor groups of pixels that are non-distinct or non-identifiable to aviewer, when viewed outside of the context of their surrounding pixels.

In some alternative embodiments, a feature point may comprise one ormore salient features. Salient features may refer to salient regions ordistinct (e.g., recognizable) objects within an image. As used herein,salient features may refer to salient regions or distinct (e.g.,recognizable) objects within an image. Salient features may refer toelements within an image that are likely to stand out or catch attentionof a human observer. A salient feature may have semantic meaning.Salient features may refer to elements that may be identifiedconsistently under computer vision processes. A salient feature mayrefer to animate objects, inanimate objects, landmarks, marks, logos,obstacles, and the like within an image. A salient feature may bepersistently observed under differing conditions. For example, a salientfeature may be persistently identified (e.g., by a human observer or bycomputer programs) in images acquired from different points of view,during different times of the day, under different lighting conditions,under different weather conditions, under different image acquisitionsettings (e.g., different gain, exposure, etc), and the like. Forexample, salient features may include humans, animals, faces, bodies,structures, buildings, vehicles, planes, signs, and the like.

Salient features may be identified or determined using any existingsaliency calculating methods. For example, salient features may beidentified by contrast based filtering (e.g., color, intensity,orientation, size, motion, depth based, etc), using a spectral residualapproach, via frequency-tuned salient region detection, via a binarizednormed gradients for objectness estimation, using a context-aware topdown approach, by measuring visual saliency by site entropy rate, andthe like. For example, salient features may be identified in a saliencymap that is generated by subjecting one or more images to contrast basedfiltering (e.g., color, intensity, orientation, etc). A saliency map mayrepresent areas with feature contrasts. A saliency map may be apredictor where people will look. A saliency map may comprise a spatialheat map representation of features or fixations. For example, in asaliency map, salient regions may have a higher luminance contrast,color contrast, edge content, intensities, etc than non-salient regions.In some embodiments, salient features may be identified using objectrecognition algorithms (e.g., feature based methods, appearance basedmethods, etc). Optionally, one or more objects or types of patterns,objects, figures, colors, logos, outlines, etc may be pre-stored aspossible salient features. An image may be analyzed to identify salientfeatures that are pre-stored (e.g., an object or types of objects). Thepre-stored salient features may be updated. Alternatively, salientfeatures may not need to be pre-stored. Salient features may berecognized on a real time basis independent to pre-stored information.

FIG. 14 illustrates a vision sensing system comprising a plurality ofbinocular cameras and at least one monocular camera, in accordance withsome embodiments. A plurality of vision sensors #1, 2, 3, 4, 6, 7, 8, 9and 10 are supported on different sides (e.g. front, rear and lateralsides) of the vehicle body. The vision sensors may have a sameorientation or different orientations relative to the side of thevehicle to which they are mounted. For example, vision sensors #1, 2, 8and 9 may be oriented at a predetermined angle (e.g. about 45 degrees)relative to the left and right sides of the vehicle body. Additionallyor optionally, vision sensors #3 and 7 may be oriented at anotherpredetermined angle (e.g. about 135 degrees) relative to the front sideof the vehicle body. In some embodiments, the vision sensors may berigidly coupled to the vehicle in a preset orientation. In otherembodiments, the vision sensors may be capable of rotating about avertical axis to change their orientations as described elsewhereherein. The change in orientations (optical axes) of the vision sensorsmay occur when the vehicle is moving or when the vehicle is stationary.

Each of the above vision sensors can be used collectively with one ormore other vision sensors to form one or more binocular cameras. Thevision sensors can be combined in different ways to form differentbinocular cameras for near and long range visual sensing, as describedelsewhere herein. Accordingly, a plurality of binocular cameras can beconfigured and used to detect objects located at various distances frommultiple sides of the vehicle.

In some embodiments, a binocular camera B1 can be formed by collectivelyutilizing vision sensors #1 and #3. Similarly, a binocular camera B2 canbe formed by collectively utilizing vision sensors #1 and #2.Accordingly, a plurality of binocular cameras may be provided ondifferent sides of the vehicle by combining the vision sensors indifferent configurations, as shown in the following table.

Vehicle body Binocular Visual detection Combination side camera range ofvision sensors Left B1  Far #1 and #3 B2  Intermediate #1 and #2 B3 Near #2 and #3 Right B4  Far #7 and #9 B5  Intermediate #8 and #9 B6 Near #7 and #8 Front B7  Far #3 and #7 B8  Intermediate #3 and #6 B9 Intermediate #4 and #7 B10 Near #3 and #4 B11 Near #4 and #6 B12 Near #6and #7 Rear B13 Intermediate #0 and #10

In some embodiments, different sets or combinations of binocular camerasmay be selectively activated as the vehicle is moving through differenttypes of environment (e.g., indoor, outdoor, densely-built areas, openareas, different terrains, altitudes, etc.). The binocular cameras maybe selectively used or activated depending on the type of environment inwhich the vehicle is operating. For example, when the vehicle is movingthrough an environment that has a high object density (e.g., in adensely populated city), binocular cameras having near and/orintermediate sensing ranges may be selectively activated. Sincesurrounding objects are likely to be closer to the vehicle, there maynot be a need to activate binocular cameras that have far sensing rangesin such an environment.

Conversely, when the vehicle is moving through an environment that has alow object density (e.g., in a sparsely populated city or open terrain),binocular cameras having far and/or intermediate sensing ranges may beselectively activated. Since surrounding objects are likely to befurther away from the vehicle in such an environment, there may not be aneed to activate binocular cameras that have near sensing ranges in suchan environment.

In addition to the binocular cameras, the vision sensing system may alsocomprise a forward-facing monocular camera #5 mounted on the front sideof the vehicle, and rearward-facing monocular camera #11 mounted on therear side of the vehicle. The monocular cameras may be similar thosedescribed in FIG. 13. For example, the forward-facing monocular camera#5 may have a higher imaging resolution (4K) and the rearward-facingmonocular camera #11 may have a lower imaging resolution (1080p). The 4Kimage data can be used to detect objects that lie within a maximumpredetermined distance (e.g. 200 m) from the front side of the vehicle.The 1080p image data can be used to detect objects that lie withinanother maximum predetermined distance (e.g. 100 m) from the rear sideof the vehicle. Accordingly, the forward-facing monocular camera #5 canbe used for monitoring the environment in front of the vehicle, and therearward-facing monocular camera #11 can be used for monitoring theenvironment behind the vehicle. The 4K and 1080p image data can bepolychromatic (e.g., RGB, CMYK, HSV) or monochromatic (e.g., grayscale,black-and-white, sepia). In some embodiments, one or more of themonocular cameras (e.g., forward-facing monocular camera) can beconfigured to capture color image data. Color images contain moreinformation compared to monochromatic images. For example, differentcolors can aid in detecting the type and physical nature of objects(e.g., children, toy dolls, statues, types of animals, color of theanimals, etc.). Objects such as pedestrians, other vehicles, man-madestructures and obstacles can be detected by processing the image datausing one or more visual detection algorithms as described elsewhereherein. In some embodiments, the color image data may be processed usingone or more processors implementing an Artificial Neural Network (ANN),which can be trained to more accurately identify the nature of objectsprogressively over time, determine whether the objects pose a potentialdanger and whether human lives (either the driver's or people outside ofthe vehicle are in danger), etc.

FIG. 15 illustrates a vision sensing system on a vehicle in accordancewith some embodiments. The system of FIG. 15 is similar to that of FIG.10 except it further includes a forward-facing monocular camera 138-1mounted on a top portion (e.g. hood) of the vehicle, along with theplurality of near and far sensing binocular cameras located on differentsides (e.g., front, rear and lateral sides of the hood) of the vehicle.

In some embodiments, the sensing assembly may include a radar system.The radar system may be a wave imaging radar system. The radar systemmay be configured to operate at millimeter wavelengths (e.g. 1 cm to 1mm), at frequencies ranging from 30 GHz to 300 GHz. Electromagneticwaves at millimeter wavelengths are not completely attenuated bysubstantial distances of fog or smoke, compared to light in the visiblespectrum. Also, electromagnetic waves at millimeter wavelengths canpenetrate clothing and significant thickness of materials such as drywood and wallboard. Accordingly, the radar system can improve navigationand visibility through environments with thick fog or smoke. The radarsystem has a stable detection performance that is independent of coloror texture of the object surface, and has excellent object penetrationability (e.g. ability to penetrate through rain, fog, smoke and certaintypes of materials). The detection accuracy of the radar system is notsignificantly affected by the surrounding environment and by weatherconditions. The radar system may have a detection range of about 100 m,110 m, 120 m, 130 m, 140 m, 150 m, 160 m, 170 m, 180 m or more, and iscapable of operating in darkness. The radar system can be used tomonitor a wide environmental area and can be used in conjunction withlidar system and vision sensing system for sensing the surrounding ofthe vehicle.

FIG. 16 illustrates a radar system being arranged on a vehicle inaccordance with some embodiments. The radar system may include one ormore millimeter-wave radar units. For example, the radar system mayinclude a first radar unit 142-1 supported on a front side of thevehicle and configured to detect objects in front of the vehicle.Additionally or optionally, the radar system may include a second radarunit 142-2 supported on a rear side of the vehicle to detect objectsbehind the vehicle.

In some embodiments, the radar system may be configured to modulate amillimeter wave signal with a two or more lower frequency signals (knownas frequency shift keying or FSK) or with a linearly changing (rampingup or ramping down in frequency) lower frequency signals (known aslinear frequency modulated LFM). The radar system can measure thedistance to a target and the relative velocity of the targetsimultaneously. The radar system may aid in autonomous vehicle controland vehicle collision avoidance.

The radar system may include one or more circuits that transmit andreceive millimeter waves. The radar system may be supported on a portionof the vehicle (e.g. behind a portion of the vehicle's hood) that isconfigured to be transparent to millimeter-wave energy.

The radar system may include an antenna, a millimeter-wave unit, and asignal processing unit. The antenna may be a planar antenna which isadvantageous for reducing size and thickness. For example, the antennamay be about 10 cm in diameter, and can be supported behind a vehicle'sgrill or fender, or designed into the front portion of the vehicle.

The millimeter-wave unit may include a monolithic microwave integratedcircuit based on high electron mobility transistors (HEMTs) used in thetransmission/reception section. In some embodiments, the millimeter-waveunit may include a voltage-controlled oscillator (VCO) that employs amicrowave IC (MIC). The VCO may be configured to receive a triangularmodulation wave to produce an FM-modulated 30-GHz signal. Themillimeter-wave unit may also include a transmission module forincreasing the frequency of the signal, and amplifying the signal to apredetermined level for transmission. The reception module is configuredto capture and amplify the signal reflected from a target.

The signal processing unit may be configured to process theamplified/reflected signals from the reception module. The signalprocessing unit may be configured to detect the difference between therelative velocity obtained through range changes and the measuredrelative velocity. When the difference exceeds a predeterminedthreshold, the signal processing unit may consider the target anunwanted reflection and discard the data. The signal processing unit mayalso monitor the continuity of the range and velocity data. When thesignal processing unit detects continuity for a target a plurality oftimes, the signal processing unit may then determine that the radarsystem has detected a true target and thus stores the data. Thecontinuity of the target may be judged by comparing to that of aprevious target. In some embodiments, the range and relative velocityobtained by the signal processing unit may be displayed on a separatedisplay unit as target data.

In some embodiments, the sensing assembly may include an ultrasonicsensing system for proximity sensing. FIG. 19 illustrates an ultrasonicsensing system being arranged on a vehicle in accordance with someembodiments. The ultrasonic sensing system 150 may include a pluralityof ultrasonic sensors supported on different sides (e.g. front, rear,and lateral sides) of the vehicle body. The ultrasonic sensors mayinclude, for example a first set of ultrasonic sensors 152-1 and 152-2located at or near the front side of the vehicle, a second set ofultrasonic sensors 154-1 and 154-2 located at or near the rear side ofthe vehicle, a third set of ultrasonic sensors 156-1 and 156-2 locatedat or near the left side of the vehicle, and a fourth set of ultrasonicsensors 158-1 and 158-2 located at or near the right side of thevehicle. The ultrasonic sensors may be located near or adjacent to thevision sensors. The ultrasonic sensors can be situated on a portion ofthe vehicle that is different from the portions used to carry the visionsensors. The ultrasonic data may be used to supplement the visualcorrelation of image data to identify invalid pixel points. For example,image data captured by binocular cameras in the vision sensing systemmay not be useful for detecting the position of a white-colored wall, ora glass wall. In contrast, ultrasonic data collected by the ultrasonicsensing system 150 can be used to detect the position/distance ofobjects having no obvious texture or that are transparent. Ultrasonicsensors may be configured to detect objects independent of visualcharacteristics, such as color, reflectivity, or texture. Ultrasonicsensors may be capable of detecting objects that are not capable ofbeing detected by vision sensors.

Similar to the radar system, the ultrasonic sensors can operate reliablyin harsh environments, such as in dirt, dust, or fog environments. Theultrasonic sensors are capable of detecting small targets or objects.Advantages of ultrasonic sensors include compact form factor and easyinstallation. The ultrasonic sensors can be used to detect proximalareas to the vehicle that do not fall within the sensing scope of theother sensing systems (i.e. “blind” spots).

In some embodiments, the ultrasonic sensing system 150 may include atleast two ultrasonic sensors provided at the front, rear, left and rightsides of the vehicle. An effective detection range of the ultrasonicsensors may be a distance up to 1 m, 2 m, 3 m, 4 m, 5 m, 6 m, 7 m, 8 mor more from each side of the vehicle. The ultrasonic sensors may beoperably coupled to the vehicle via one or more carriers, that permitthe ultrasonic sensors to move relative to the vehicle with respect toup to six degrees of freedom. For example, an ultrasonic sensor may beconfigured to tilt (e.g. pitch upwards, downwards or sideways) by apredetermined amount, thereby changing the direction and scope of scanrelative to the vehicle. In some embodiments, the carriers may includeone or more motors configured to rotate and/or translate the ultrasonicsensors in one or more degrees of freedom. The ultrasonic sensors can beactuated and controlled to scan different areas proximal to the vehicleso as to avoid any “blind” spots. In some embodiments, a plurality ofcarriers may be configured to control the positions of the ultrasonicsensors, for example by rotating and/or translating the ultrasonicsensors simultaneously or sequentially, to sense an entire area proximalto the different sides of the vehicle. In some alternative embodiments(not shown), the ultrasonic sensors may be rigidly coupled to the sidesof the vehicle.

FIG. 17 illustrates how one or more sensors may be configured to changeorientation based on a vehicle's motion or predicted motion inaccordance with some embodiments. In some embodiments, a vehicle 100 maycomprise a sensing assembly 110. The sensing assembly may comprise oneor more sensors having corresponding detectable ranges 202, 204, 206.The one or more sensors of the sensing assembly may be of the same typeof sensors or different types of sensors.

In one example, when a vehicle is moving in a forward direction, one ormore of the sensors of the sensing assembly may be oriented in a forwardfacing direction, as illustrated in Part A. The one or more sensors ofthe sensing assembly may be oriented in a direction of travel. In someembodiments, if the vehicle is traveling backwards, the one or moresensors may be oriented in a rear facing direction or another direction.In some embodiments, the one or more sensors may remain facing in aforward direction when the vehicle is traveling directly backwards. Thevehicle may rely on one or more other sensors when traveling backwards.When the vehicle is turning to the right or left, the one or moresensors may remain facing forward, or may be re-oriented to face thedirection of the turn.

One or more sensors of a sensing assembly may be configured to changeits orientation based on the vehicle's motion or predicted motion path.In one example, one or more sensors that face forward when the vehicleis moving in a forward direction may be configured to change itsorientation based on the vehicle's motion or predicted motion path.Optionally, all of the sensors that face forward when the vehicle ismoving in a forward direction may be configured to change itsorientation based on the vehicle's motion or predicted motion path.Alternatively, at least one of the sensors may remain forward facingeven when the vehicle turns, while one or more other sensors may changeits orientation based on the vehicle's motion or predicted motion path.

The orientation of the one or more sensors may change based on thevehicle's motion or predicted motion path. The orientation may changebased on the vehicle turning or being predicted to turn. The orientationmay change based on the vehicle location. For instance, the orientationmay change because the intersection is recognized, regardless of whetherthe vehicle will go straight or turn. The orientation may change inreal-time. For instance, the orientation changing in real-time maycomprise the orientation starting to change within 15 seconds, 10seconds, 5 seconds, 3 seconds, 2 seconds, 1 second, 0.5 seconds, 0.3seconds, 0.1 seconds, 0.05 seconds, 0.01 seconds, 0.005 seconds, or0.001 seconds of a recognition of a condition that triggers the changein orientation.

The orientation of the one or more sensors may change to scan forobstacles prior to or as the vehicle is undergoing a condition thattriggers the change in orientation. For instance, the condition thattriggers the change in orientation may be the vehicle changing itsmotion or predicted motion path. A change in direction may constitute achange in motion. A predicted change in direction may constitute apredicted change in motion path. In another instance, the condition thattriggers the change in orientation may be the vehicle approaching orpassing through an intersection. The condition that triggers the changein orientation may be the vehicle approaching or entering a trafficcircle, merging onto a road, changing lanes, approaching a pedestriancross-walk, parking, entering a structure, or other maneuvers by thevehicle.

A detected change in motion or predicted change in motion may be basedon a vehicle input, data collected by sensors, and/or a map of anenvironment that the vehicle is located.

An example of a vehicle input may comprise a pre-planned driving route.The pre-planned driving route may have a start point and a destination.The vehicle may be autonomously driving along the driving route.Alternatively, a driver may drive along the driving route and/ormanually assisting the vehicle that may be semi-autonomously drivingalong the driving route. The driver may be manually driving along thedriving route and the vehicle may provide driving assistance as thedriver drives along the route. Another example of a vehicle input maycomprise activation of a turn signal of the vehicle. For instance, if adriver or the vehicle autonomously turns the turn signal on, adetermination may be made that the vehicle is about to turn or maneuverin the direction indicated by the turn signal. A vehicle input maycomprise a rotation of a steering wheel of the vehicle. The rotation ofthe steering wheel may be initiated by a driver, or autonomously by thevehicle. The direction that the steering wheel is rotated may beindicative of the direction that the vehicle is starting to turn. Thedegree that the steering wheel is rotated may be indicative of the arclength or sharpness of turn. The vehicle input may comprise a change indirection of one or more driving wheels of the vehicle. The drivingwheels may or may change direction in response to a rotation of thesteering wheel. The driving wheels may change direction withoutrequiring rotation of the steering wheel. The direction that the drivingwheel is rotated may be indicative of the direction that the vehicle isstarting to turn. The degree that the driving wheel is rotated may beindicative of the arc length or sharpness of turn.

Data collected by sensors may also be used to detect a condition thatmay trigger orientation of one or more sensors. For example, one or moresensors on-board the vehicle may be used to detect and/or recognize thatthe vehicle is approaching an intersection. Data about the intersection(e.g., number of roads intersecting, level of traffic, direction ofroads) may be collected with aid of the sensors. Other features, such astraffic lights, traffic circles, merges, lanes splitting off, pedestriancross-walks, barricades, construction, signs for re-direction, ramps,debris, potential obstacles, or other features may be detected with aidof one or more sensors. In another example, the one or more sensorson-board the vehicle may be used to detect and/or recognize the changeof orientation in the road. For example, the road may curve to the rightor left. The sensors may be used to detect the direction and/or degreeof curve.

A map of the environment within which the vehicle is located may also beused to detect a condition that may trigger orientation of one or moresensors. For example, the location of the vehicle on the map may bedetermined. The map may include positioning of roads and/or updatedinformation about particular areas (e.g., whether there is an accident,construction, broken traffic light, etc.). For example, the map may beused to detect and/or recognize that the vehicle is approaching anintersection. Data about the intersection (e.g., number of roadsintersecting, level of traffic, direction of roads) may be known on themap. Other features, such as traffic lights, traffic circles, merges,lanes splitting off, pedestrian cross-walks, barricades, construction,or other features may be on the map. In another example, the map may beused to predict the change of orientation in the road. For example, theroad may curve to the right or left. The map may have information aboutthe direction and/or degree of curve.

In some embodiments, the degree of the change in orientation may dependon the motion or predicted motion of the vehicle. For instance, thedegree of change in orientation may depend on the angle at which thevehicle changes orientation. In one example, the degree of change oforientation may be greater when there is a greater predicted change inorientation. For example, a sensor may turn further to the left when thevehicle is making a steep turn to the left, as compared to a smallerturn to the left when the vehicle path is curving a little to the left.There may be a linear relationship, exponential relationship, or anyother type of relationship between the degree that the sensor turns andthat the vehicle turns or is predicted to turn. The angle of rotation ofthe sensor may be adjusted based on a turn angle or arc length of thevehicle's motion or predicted motion path.

The speed at which the sensor changes orientation may depend on themotion or predicted motion of the vehicle. For instance, the speed ofchange in orientation may depend on the angle at which the vehiclechanges orientation, or the sharpness of the turn. In one example, thespeed of change of orientation may be greater when there is a greaterpredicted change in orientation. For example, a sensor may turn to theleft faster when the vehicle is making a steep turn to the left, ascompared to a slower turn to the left when the vehicle path is curving alittle to the left. The sensor may turn faster if there is greatersuddenness or sharpness to the turn. The sensor may turn faster when thecurvature of the turn is higher. There may be a linear relationship,exponential relationship, or any other type of relationship between thespeed that the sensor turns and that the vehicle turns or is predictedto turn. The velocity of rotation of the sensor may be adjusted based ona turn angle or arc length of the vehicle's motion or predicted motionpath. Alternatively, the speed at which the sensor changes may beconstant or predetermined.

In some embodiments, the degree and/or speed of change may bepredetermined. A sensor may be configured to change its orientation byrotating a predetermined amount based on the vehicle's motion orpredicted motion path.

The sensors may change orientation by rotating about a yaw axis. The yawof the sensors may be altered. The sensors may rotate to the left orright when the vehicle is turning to the left or right, or is predictedto turn left or right. In some embodiments, the sensors may changeorientation by rotating about a pitch axis. The sensors may angle up ordown as the vehicle is traveling up or down a hill, or as the predictedpath change vertical orientation. The sensors may change orientation byrotating about a combination of the yaw and pitch axis. Alternatively orin addition, sensors may or may not rotate about a roll axis.

In some instances, at least one of the sensors may be configured torotate clockwise (e.g., about a yaw axis) prior to, or as the vehiclechanges its direction to the right, so as to detect a region to thefront-right of the vehicle. In another instance, at least one of thesensors may be configured to rotate counterclockwise (e.g., about a yawaxis) prior to, or as the vehicle changes its direction to the left, soas to detect a region to the front-left of the vehicle.

Part A illustrates a vehicle 100 traveling in a forward direction with aplurality of sensors facing forward. The detectable ranges 202, 204, 206of the forward-facing sensors may be provided in front of the vehicle.Additional sensors having various detectable ranges may be provided, asdescribed elsewhere herein.

Part B illustrates the vehicle approaching an intersection. Theorientation at of at least one of the sensors may change whenapproaching or upon reaching the intersection. In some embodiments, afirst sensor may still face forward, so that a detection range 202 is infront of the vehicle. A second sensor may turn towards the right, sothat the detection range 204 is to the right-front of the vehicle. Athird sensor may turn towards the left, so that the detection range 206is to the left-front of the vehicle. Such a change in orientation may bemade prior to any determination on which direction the vehicle will goat the intersection. Such a change in orientation can be made after andbased on a determination or detection of which direction the vehiclewill go at the intersection. In one example, it may be known the vehicleis turning or will turn to the left (e.g., moving from Position A toPosition B). The sensors may change orientation in response to knowingthe vehicle will turn to the left. The forward facing sensor may beuseful for detecting conditions in front of the vehicle. Theright-facing sensor may be useful for detecting oncoming traffic orother conditions to the right of the vehicle when the vehicle makes theleft turn. The left-facing sensor may be useful for detecting conditionsto the left of the vehicle, such as oncoming traffic from the left,potential obstacles, or other conditions. In some embodiments, thesensor remaining to the front, turning to the left, and turning to theright, may all be the same type of sensor. The sensors may have the samedetectable range sizes (e.g., ranges, widths, and/or shapes).Alternatively, they may have different detectable ranges (e.g., ranges,widths, and/or shapes). The sensor remaining to the front, turning tothe left, and turning to the right may comprise two or more differenttypes of sensors. The same or different types of sensors may be anysensor type as described elsewhere herein.

In some embodiments, one or more of the sensors may change orientationto face the direction that the vehicle is turning. In one example, allof the sensors may change orientation to face the direction that thevehicle is turning. In some instances, one of the sensors may remainfacing to the front while the remaining sensors may change orientationto face the direction that the vehicle is turning. In another instance,one or more sensors may change orientation to face a different directionfrom the direction the vehicle is turning. For instance, if the vehicleis turning to the left, one or more sensors may turn to face to theright, or if the vehicle is turning to the right, one or more of thesensor may turn to face to the left. This may be in addition to, or asan alternative to one or more of the sensors facing forward and/or inthe direction that the vehicle is turning. As previously described, thesensors that remain facing in the same orientation or change orientationmay be of the same sensor type or different sensors types.

In some embodiments, the sensors may re-adjust orientation to face thevarious number of roads presented before the vehicle. For instance, if avehicle comes to a four-way intersection, the sensors may face in threedifferent directions, to be oriented towards each of the other threedirections that the vehicle may travel. If the vehicle comes to athree-way intersection, the sensors may face in two differentdirections. If the vehicle is merely following a path that is curvingwith no offshoots or directions that the vehicle might turn, the sensorsmay be oriented in front of the vehicle, or may be angled to follow theroad that the vehicle is traversing.

FIG. 18 provides an additional illustration of how one or more sensorsmay be configured to change based on a vehicle's motion or predictedmotion in accordance with embodiments. A vehicle 100 may be travelingalong a predicted path. One or more sensors of a sensing assemblyon-board the vehicle may have various detectable ranges 202, 204, 206.The sensors may be of the same type or of different types. The sensorsmay have the same detectable ranges (e.g., detectable range, width,and/or shape) or may have different detectable ranges.

One or more sensors may have an orientation relative to the vehiclebased on a vehicle's motion or predicted motion of travel. For example,if the vehicle is predicted to move forward, a sensor may remain forwardfacing with a detectable range 202 in front of the vehicle. If thevehicle is predicted to turn towards the right, the sensor may turn tothe right with a detectable range 202 to the right and front of thevehicle. One or more of the sensors may face a direction of travel (orpredicted direction of travel) of the vehicle.

In some embodiments, one or more sensors may face at a differentdirection from the direction of travel. The one or more sensors mayreorient to face at a different direction from the direction of travel.For example, one or more sensors may face toward the right to have adetectable range 204 to the right of the direction of travel, and/or oneor more sensors may face toward the left to have a detectable range 206to the left of the direction of travel. These sensors may remain facingat different directions from the direction of travel or may at timesalign with the direction of travel.

The one or more sensors may orient themselves to face the variousdifferent directions. The sensors may start in a forward direction andorient themselves to the various different directions as needed. Thesensors may directly move from one desired orientation to another.

In some embodiments, the sensors may scan back and forth betweendifferent directions. The sensors may rotate back and forth at apredetermined speed. The sensors may rotate back and forth at a speeddepending on direction of travel or curvature of path. In someinstances, a sensor may scan between a forward facing direction and apreset direction to the left or right. In some instances, the sensor mayscan between a direction of travel and a preset direction relative tothe vehicle or preset degree relative to the direction of travel, to theleft or right. In some embodiments, the direction to the left or rightof the vehicle may depend on a trigger condition that causes the sensorsto scan back and forth.

In addition to changing orientation about the yaw axis, the sensors maybe capable of changing orientation about a pitch axis. In someembodiments a predicted path of the vehicle may be for the vehicle to gofrom downhill to uphill. The pitch of one or more sensors may be altereddepending on the predicted vertical road change. For example, when avehicle is traveling on a flat road, the sensors may have a neutralhorizontal position. If the vehicle is cresting a hill, the sensors mayangle slightly downwards to capture more of the road in the detectablerange. If the vehicle is at, or entering, a trough or valley of theroad, the sensors may angle slightly upwards to capture more of the roadat the detectable range. Any description herein relating to horizontalchange of orientation may also apply to vertical change of orientation.For instance, descriptions relating to change of orientation, speed oforientation, sensor types, as described elsewhere herein apply tovertical orientation as well.

FIG. 20 illustrates a sensing system controller in communication with asensing assembly of a vehicle, in accordance with some embodiments. Asensing system controller 200 may be operably connected to two or moresame or different types of sensors in a sensing assembly 110. Forexample, the sensing system controller may be in communication withlidar 120, cameras 130, radar 140, ultrasonic sensors 150, GPS 160and/or odometers 170. In some cases, the sensing system controller maybe communication with N different types of sensors, where N can be anyinteger greater than one. The sensing system controller can comprise oneor more processors that are configured to obtain sensing data collectedfrom a plurality of sensors coupled to a vehicle (e.g., vehicle 100).

The sensing system controller can be configured to process the sensingdata by fusing two or more sets of data from different sensors. Forexample, the sensor data from two or more groups of single-channel lidarunits and stereo cameras can be fused into a set of RGB-D data. Thesensing system controller may analyze the RGB-D data to detect obstaclesin the environment, and provide the obstacle information to a vehiclenavigation controller 300. The sensing system controller may alsoinclude one or more processors for processing (e.g. fusing) data fromthe forward and/or rear monocular cameras, long range lidar, and/ormillimeter wave radar, so as to implement remote object monitoring andobstacle avoidance. The sensing system controller may analyze data fromone or more ultrasonic sensors to detect obstacles in an area proximateto the vehicle, that may lie in the “blind” spots of the other sensors.The sensing system controller may also provide data from the vehicleodometer and GPS to the vehicle navigation controller 300 to assist inposition, driving and path planning.

The sensor fusion approaches described above can be applied to varioustypes of functionalities, including navigation, object recognition, andobstacle avoidance. In some embodiments, environmental data obtainedusing sensor fusion results can be used to improve the robustness,safety, and flexibility of operation of the vehicle by providingaccurate location information as well as information regarding potentialobstructions. The environmental data can be provided to a user (e.g.,via remote controller or terminal, mobile device, or other user device)so as to inform the user's manual control of the vehicle. Alternativelyor in combination, the environmental data can be used forsemi-autonomous or fully autonomous control systems to direct theautomated driving of the vehicle.

The obstacles described herein may be substantially stationary (e.g.,buildings, plants, structures) or substantially mobile (e.g., humanbeings, animals, vehicles, or other objects capable of movement). Someobstacles may include a combination of stationary and mobile components(e.g., a windmill). Mobile obstacles or obstacle components may moveaccording to a predetermined or predictable path or pattern. Forexample, the movement of a car may be relatively predictable (e.g.,according to the shape of the road). Alternatively, some mobileobstacles or obstacle components may move along random or otherwiseunpredictable trajectories. For example, a living being such as ananimal may move in a relatively unpredictable manner. Different types ofenvironments may be associated with different amounts and types ofobstacles. For example, a high altitude environment may have few or noobstacles. In contrast, an indoor environment or a low altitudeenvironment may have more obstacles. Some types of low altitude, outdoorenvironments (e.g., fields and other flat, open spaces) may have fewerobstacles than other types (e.g., urban settings and other highlypopulated areas, forests). Accordingly, a vehicle described hereinoperating within an environment with a high obstacle density may beexposed to an increased risk of collisions, near-misses, or other safetyincidents. Conversely, vehicle operation within a low obstacle densityenvironment may be relatively safe. Mobile obstacles may pose anincreased risk compared to stationary obstacles, as mobile obstacles maycollide with or obstruct the vehicle independently of any action takenby the vehicle. The sensor fusion approaches can be used to improvedetection of obstacles within the environment in which the vehicleoperates.

The vehicle described herein can be configured to move along differentmotion paths between a plurality of locations. For many real worldapplications, knowing merely the position and motion of the vehicle maynot be sufficient for real-time navigation. For example, the surroundingenvironment may include obstacles in the path between the vehicle and atarget destination. These obstacles may be stationary, capable ofmovement, or in motion. As such, information about the externalenvironment may be necessary for the vehicle to avoid such obstacles byre-planning its path in real-time. In some embodiments, informationabout the external environment may be provided in a 3D map based on oneor more images captured by cameras and other sensors onboard thevehicle. A motion path for the vehicle can be generated by using the 3Dmap.

The embodiments disclosed herein can be used to perform obstacleavoidance maneuvers in order to prevent a vehicle from colliding withenvironmental objects. In some embodiments, obstacle detection andavoidance can be automated, thereby improving safety and reducing userresponsibility for avoiding collisions. This approach may beadvantageous for inexperienced operators as well as in situations wherethe user cannot readily perceive the presence of obstacles near thevehicle. Additionally, the implementation of automated obstacleavoidance can reduce the safety risks associated with semi-autonomous orfully autonomous vehicle navigation. Furthermore, the multi-sensorfusion techniques described herein can be used to generate more accurateenvironmental representations, thus improving the reliability of suchautomated collision prevention mechanisms.

FIG. 21 illustrates an automatic driving system 1900 comprising ahardware sensor module 1910, a sensing module 1920, and a navigation andposition module 1930, in accordance with some embodiments. The hardwaresensor module may correspond to the sensing assembly described elsewhereherein. For example, the hardware sensor module may include a firstgroup of sensors 1912 for proximity detection, and a second group ofsensors 1914 for long-distance detection. The first group of sensors mayinclude, for example one or more ultrasonic sensors coupled to multiplesides of the vehicle. The ultrasonic sensors may be configured to adjustits orientation to change its direction and scan area, so as toadequately sense the area proximate or surrounding the vehicle. Theultrasonic sensors can also scan areas that lie in the “blind” spots ofthe second group of sensors. The second group of sensors may include,for example a forward monocular camera, a long range lidar, and amillimeter-wavelength radar as described elsewhere herein. The sensormodule may also include multiple lidar units 1916 provided in groupslocated around the vehicle, and multiple binocular cameras 1918. Thelidar units may be single-channel lidar units. Additionally andoptionally, the sensor module may include an odometer 1919-1 and a GPSsensor 1919-2.

The sensing module 1920 may receive the data from the sensor module 1910in an asynchronous and Publish/Subscribe manner. The sensing module mayinclude one or more submodules that subscribe to data collected by thesensor module 1910. The sensors in the sensor module 1910 may beconfigured to automatically send data to the corresponding subscribedsub-modules. For example, the ultrasonic sensors in the first group ofsensors may send data to a submodule 1922 for detecting objects that areproximal to the vehicle. Similarly, the forward monocular camera, longrange lidar, and millimeter-wavelength radar in the second group ofsensors may send data to another submodule 1924 for detecting objectsthat are distant to the vehicle.

The multiple lidar units and binocular cameras may send data to asubmodule 1926 which fuses the data into a set of RGB-D data. The RGB-Ddata is then provided to a detection submodule 1928 configured toanalyze the RGB-D data to detect obstacles in the environment.

One or more submodules in the sensing module 1920 may be configured toprovide the obstacle detection information to the navigation andposition module 1930. The navigation and position module may comprise aposition submodule 1932 for determining a plurality of positions andpathways for avoiding the obstacles so as to safely navigate the vehiclewithin an environment, based on the obstacle information obtained by thesensing module, along with real-time data from the vehicle odometer andthe GPS sensor. The position submodule then sends the plurality ofpositions and pathways to a navigation submodule 1934 configured tocontrol autonomous driving of the vehicle based on the plurality ofpositions and pathways.

In some embodiments, a plurality of sensors may be used to collectinformation about a vehicle. The plurality of sensors may comprise aplurality of sensor types, such as any combination of the various sensortypes described elsewhere herein. For example, a first sensor type maybe a vision sensor while a second sensor type may be lidar units.However, any combination of different sensor types may be provided.

In some embodiments, data from a first sensor type may be fused withdata from a second sensor type. The fused data from the first and secondsensor types may be subsequently used for detection about a vehicle.

In another example data from the first sensor type may be used fordetection about a vehicle, and data from a second sensor type may beused for detection about a vehicle. This detection data may then befused to come up with a master detection about a vehicle.

In some embodiments, weights may be assigned to data from the firstsensor type and/or the second sensor type. The weight might depend onsuitability of the sensor type for operation within an environment typewithin which the vehicle is operating. For example, if a first sensortype is more suitable for the environment that the vehicle is operatingthan the second sensor type, then the data from the first sensor typemay be weighted more than the data from the second sensor type. In someembodiments, if data from a particular sensor type is not at all suitedfor operation within the environment, the data from the sensor type maybe weighted at zero, or another low value. The data may be weightedprior to or after fusing the data. The data may be weighted prior to orafter detection about the vehicle.

For any number of data from any number of sensor types (e.g., 1, 2, 3,4, 5, 6, 7, 8, or more), the data may be fused before detection occurs,or afterwards. Data from any number of sensor types may be weighted.They may all be weighted together at the same time, or sequence.

The data from the lidar sensors (e.g. single or subsets of lidar units)and the data from vision sensors (e.g., binocular cameras or monocularcameras) can be processed using at least one or more of the followingsensor fusion techniques. In some embodiments, the data from the lidarsensors and the vision sensors may be first fused together, and then thefused data can be used for obstacle detection and/or recognition of thesurrounding environment.

In some other embodiments, some portion of the obstacle detection mayoccur prior to sensor fusion. For example, the data from the lidarsensors may be individually used to detect obstacles, and the data fromthe vision sensors may be individually used to detect obstacles. Theobstacles may or may not be the same, of the same type, or at the samedistance from the vehicle. An environmental map may then be created byfusing together information about the obstacles detected respectively bythe lidar sensors and the vision sensors. One or more processors may beconfigured to determine characteristics of the obstacles from theenvironmental map, and correlate or corroborate the obstacles detectedby the respective lidar sensors and vision sensors. In some embodiments,the one or more processors may be configured to determine if the datafrom each of the lidar sensors and vision sensors is reliable. Forexample, under foggy conditions, if visibility is determined to fallbelow a threshold based upon image data collected by the vision sensors,the data from the vision sensors may be deemed unreliable, and may notbe used in the sensor fusion (or alternatively, assigned a low weight inthe sensor fusion). Under these foggy conditions, only the data from thelidar sensors may be used for obstacle detection. This may correspond toa first condition in which the data from the vision sensors is assigneda weight of 0 (minimum), whereas the data from the lidar units isassigned a weight of 1 (maximum). In another example, if visibility isdetermined to be above the threshold, the data from the vision sensorscamera can then be used. Under such circumstances, the lidar sensor dataand vision sensor data can each be used to detect obstacles, and thedetected obstacles can then be fused together in an environmental map.The processors can be configured to determine a veracity (or accuracy)of the detected obstacles based on the fused data or environmental map.

In some embodiments, the data from the various sensors may be mappedonto a point cloud. One or more processors may be configured tocontinuously monitor the point cloud to determine whether points in thecloud are due to noise, or come from a same object/obstacle. Theprocessors may be configured to track one or more points, and theirspatial locations within the point cloud. For example, if the processorsis tracking a particular point or cluster of points and determine that aduration of those point(s) in the point cloud is less a predefined timethreshold, the processors may then classify those point(s) as noise andexclude them from further analysis. In some embodiments, the processorsmay assign a rating to objects in the point cloud based on the durationof their appearance in the point cloud. Objects appearing for brief timeperiods may be assigned a lower rating compared to objects appearing forlonger time periods. In some embodiments, the rating of an object canalso be determined by comparing the object from the point cloud to a setof known object models. For example, an object that closely matches aknown model may be assigned a higher rating, whereas another object thatless closely matches the known model may be assigned a lower rating. Therating of an object in the point cloud may be used to determine whetherthe object is in fact a real physical object, as well as the objecttype. In some cases, objects that have a rating greater than a firstpredetermined rating may be classified as real physical objects, whereasobjects that a rating less than a second predetermined rating may beclassified as noise data. In some embodiments, certain objects may haveratings that fall in between the first and second predetermined ratings.In those embodiments, the identification of those objects may beambiguous, and the processors may be configured to continue monitoringthose points over time to establish whether they correspond to realphysical objects or noise data. In some cases, one or more sensors maybe configured to collect more data on those objects or points that aredeemed to be ambiguous.

FIG. 22 illustrates the time synchronization of different types ofsensors in a sensor module, in accordance with some embodiments. Aspreviously described, different submodules in a sensing module 1920 cansubscribe to different sensor data from the hardware sensor module 1910,whereby each sensor sends data to the corresponding submodule(s). Asshown in FIG. 20, a plurality of system synchronization signals t_(k)and so forth may be generated at predetermined or random time intervals.The submodules may be configured to collect sensor data (e.g., ascollected data for current frame), and process the data each time asystem synchronization signal is received. The navigation and positionmodule may be configured to receive data from lidars, cameras and GPSsensors that had been processed. Each time a system synchronizationsignal (e.g., t_(k)) is received, the next sensor data t_(k) can becollected and sent to the navigation and position module. Although thesensor data may be collected at different points in time by therespective sensors, the sensor data may be fused together as a commonset of sensor data representative of the environment as at timing t_(k).

In some embodiments, a system for controlling a movable object may beprovided in accordance with embodiments. The system can be used incombination with any suitable embodiment of the systems, devices, andmethods disclosed herein. The system can include a sensing module,processing unit, non-transitory computer readable medium, controlmodule, and communication module.

The sensing module can utilize different types of sensors that collectinformation relating to the movable objects in different ways. Differenttypes of sensors may sense different types of signals or signals fromdifferent sources. For example, the sensors can include inertialsensors, GPS sensors, proximity sensors (e.g., lidar), or vision/imagesensors (e.g., a camera). The sensing module can be operatively coupledto a processing unit having a plurality of processors. In someembodiments, the sensing module can be operatively coupled to atransmission module (e.g., a Wi-Fi image transmission module) configuredto directly transmit sensing data to a suitable external device orsystem. For example, the transmission module can be used to transmitimages captured by a camera of the sensing module to a remote terminal.

The processing unit can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit can be operatively coupled to a non-transitory computerreadable medium. The non-transitory computer readable medium can storelogic, code, and/or program instructions executable by the processingunit for performing one or more steps. The non-transitory computerreadable medium can include one or more memory units (e.g., removablemedia or external storage such as an SD card or random access memory(RAM)). In some embodiments, data from the sensing module can bedirectly conveyed to and stored within the memory units of thenon-transitory computer readable medium. The memory units of thenon-transitory computer readable medium can store logic, code and/orprogram instructions executable by the processing unit to perform anysuitable embodiment of the methods described herein. For example, theprocessing unit can be configured to execute instructions causing one ormore processors of the processing unit to analyze sensing data producedby the sensing module. The memory units can store sensing data from thesensing module to be processed by the processing unit. In someembodiments, the memory units of the non-transitory computer readablemedium can be used to store the processing results produced by theprocessing unit.

In some embodiments, the processing unit can be operatively coupled to acontrol module configured to control a state of the movable object. Forexample, the control module can be configured to control the propulsionmechanisms of the movable object to adjust the spatial disposition,velocity, and/or acceleration of the movable object with respect to sixdegrees of freedom. Alternatively or in combination, the control modulecan control one or more of a state of a carrier, payload, or sensingmodule.

The processing unit can be operatively coupled to a communication moduleconfigured to transmit and/or receive data from one or more externaldevices (e.g., a terminal, display device, or other remote controller).Any suitable means of communication can be used, such as wiredcommunication or wireless communication. For example, the communicationmodule can utilize one or more of local area networks (LAN), wide areanetworks (WAN), infrared, radio, WiFi, point-to-point (P2P) networks,telecommunication networks, cloud communication, and the like.Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication modulecan transmit and/or receive one or more of sensing data from the sensingmodule, processing results produced by the processing unit,predetermined control data, user commands from a terminal or remotecontroller, and the like.

The components of the system can be arranged in any suitableconfiguration. For example, one or more of the components of the systemcan be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. In some embodiments, one or more of theplurality of processing units and/or non-transitory computer readablemedia can be situated at different locations, such as on the movableobject, carrier, payload, terminal, sensing module, additional externaldevice in communication with one or more of the above, or suitablecombinations thereof, such that any suitable aspect of the processingand/or memory functions performed by the system can occur at one or moreof the aforementioned locations.

As used herein A and/or B encompasses one or more of A or B, andcombinations thereof such as A and B. It will be understood thatalthough the terms “first,” “second,” “third” etc. may be used herein todescribe various elements, components, regions and/or sections, theseelements, components, regions and/or sections should not be limited bythese terms. These terms are merely used to distinguish one element,component, region or section from another element, component, region orsection. Thus, a first element, component, region or section discussedbelow could be termed a second element, component, region or sectionwithout departing from the teachings of the present disclosure.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” or “includes” and/or “including,” when used in thisspecification, specify the presence of stated features, regions,integers, steps, operations, elements and/or components, but do notpreclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components and/or groupsthereof.

Furthermore, relative terms, such as “lower” or “bottom” and “upper” or“top” may be used herein to describe one element's relationship to otherelements as illustrated in the figures. It will be understood thatrelative terms are intended to encompass different orientations of theelements in addition to the orientation depicted in the figures. Forexample, if the element in one of the figures is turned over, elementsdescribed as being on the “lower” side of other elements would then beoriented on the “upper” side of the other elements. The exemplary term“lower” can, therefore, encompass both an orientation of “lower” and“upper,” depending upon the particular orientation of the figure.Similarly, if the element in one of the figures were turned over,elements described as “below” or “beneath” other elements would then beoriented “above” the other elements. The exemplary terms “below” or“beneath” can, therefore, encompass both an orientation of above andbelow.

While some embodiments of the present disclosure have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the disclosure. It should beunderstood that various alternatives to the embodiments of thedisclosure described herein may be employed in practicing thedisclosure. Numerous different combinations of embodiments describedherein are possible, and such combinations are considered part of thepresent disclosure. In addition, all features discussed in connectionwith any one embodiment herein can be readily adapted for use in otherembodiments herein. It is intended that the following claims define thescope of the invention and that methods and structures within the scopeof these claims and their equivalents be covered thereby.

What is claimed is:
 1. An autonomous driving assembly for a vehiclecomprising: a plurality of lidar groups supported by a vehicle body ofthe vehicle and collectively configured to detect a periphery region inproximity to the vehicle body; wherein: different ones of the pluralityof lidar groups are supported at different areas of the vehicle body andhave different group fields of view, at least two of the different groupfields of view overlapping with each other; and each of the pluralitylidar groups includes a plurality of lidar units fixed at a samelocation, different ones of the plurality of lidar units havingdifferent unit fields of view, and at least two of the different unitfields of view overlapping with each other.
 2. The assembly of claim 1,wherein different ones of the plurality of lidar groups are supported bydifferent support structures of the vehicle body.
 3. The assembly ofclaim 1, wherein one of the plurality of lidar groups includes a firstlidar unit optically aligned in a first direction and a second lidarunit optically aligned in a second direction, an angle between the firstdirection and the second direction being about 50 degrees or less. 4.The assembly of claim 1, wherein one of the plurality of lidar groupsincludes at least three lidar units arranged in a manner to increaseoverlap between adjacent unit fields of view of the at least three lidarunits.
 5. The assembly of claim 1, wherein one of the plurality of lidargroups includes at least two lidar units having a fixed dispositionrelative to one another.
 6. The assembly of claim 5, wherein the fixeddisposition is maintained with aid of one or more linkages.
 7. Theassembly of claim 6, wherein the one or more linkages include at leastone of serial linkages or parallel linkages.
 8. The assembly of claim 5,wherein the fixed disposition is maintained with aid of a kinematiccoupling or maintained in a rigid manner.
 9. The assembly of claim 1,wherein the plurality lidar groups include a first lidar group and asecond lidar group each including at least two lidar units having afixed disposition relative to one another.
 10. The assembly of claim 9,wherein the first lidar group is configured to move relative to thesecond lidar group to adjust an overlap between a group field of view ofthe first lidar group and a group field of view of the second lidargroup.
 11. The assembly of claim 9, wherein the first lidar group andthe second lidar group are configured to move relative to each other toadjust an overlap between a group field of view of the first lidar groupand a group field of view of the second lidar group.
 12. The assembly ofclaim 9, wherein an overlap between a group field of view of the firstlidar group and a group field of view of the second lidar group isadjustable in real-time to compensate for blind spots while the vehicleis in operation.
 13. The assembly of claim 9, wherein an overlap betweena group field of view of the first lidar group and a group field of viewof the second lidar group is at least 70 degrees.
 14. The assembly ofclaim 9, wherein a group field of view of the first lidar group is atleast 160 degrees, and a group field of view of the second lidar groupis at least 160 degrees.
 15. The assembly of claim 9, wherein acollective field of view of the first lidar group and the second lidargroup is inversely proportional to a collective detection range of thefirst lidar group and the second lidar group.
 16. The assembly of claim9, wherein the at least two lidar units of the first lidar group areconfigured to not move relative to one another during operation of thevehicle, and the at least two lidar units of the second lidar group areconfigured to not move relative to one another during operation of thevehicle.
 17. The assembly of claim 9, wherein at least one of the firstlidar group or the second lidar group undergoes an initial intrinsiccalibration prior to utilization of the at least one of the first lidargroup or the second lidar group for sensing.
 18. The assembly of claim17, wherein the first lidar group does not require online calibrationduring operation of the vehicle, and the second lidar group does notrequire online calibration during operation of the vehicle.
 19. Avehicle comprising the autonomous driving assembly of claim
 1. 20. Amethod of collecting information around a vehicle for autonomousdriving, comprising: supporting, with aid of a vehicle body of thevehicle, a plurality of lidar groups of an autonomous driving assemblyfor the vehicle; and collectively detecting, by the plurality of lidargroups, a periphery region in proximity to the vehicle body to aid inautonomous driving upon coupling the autonomous driving assembly to thevehicle body, wherein: different ones of the plurality of lidar groupsare supported at different areas of the vehicle body and have differentgroup fields of view, at least two of the different group fields of viewoverlapping with each other; and each of the plurality lidar groupsincludes a plurality of lidar units fixed at a same location, differentones of the plurality of lidar units having different unit fields ofview, and at least two of the different unit fields of view overlappingwith each other.